Smart Server Defense
Powered by BlueIce3
Configuration Documentation

This page has extra information on the configuration, training, and execution of the BlueIce2 core AI, details on how to test the BlueIce2 MySQL database setup, as well as info on the configuration of the different components.
Default SSH Port: 50001 (AWS AMI Defaults to 22 automatic SSHD port checks are now disabled by default.)
Default MySQL Port: 50004 (localhost only)
Default Apache Port: 80
Checking The Base Installation
Below is a listing of the core requirements, base installation, for the BlueIce3. You can use this list to check your server configuration and make sure the requisite software is installed. Some of the packages are required for the BlueIce3 suite, some are part of an effort to make a useful bare bones system with BlueIce3 and expected server functionality ready out of the box.
The core software requirements, base installation, are as follows...
  1. Directory Structure: Check that the following directories have been create in the root directory where the install script resides.
    ./apps/python/BlueIce2: BlueIce2 AI software.
    ./db_baks: Repository for database backups.
    ./apps/python/BlueIceUtils: Automation tools for BlueIce2.
    ./apps/scripts/SmartServerDefenseCfg: Documentation, web site for BlueIce2.

    The base install directory structure is as follows.
    blueice3
    -> 0_4_0_9 : version code dir
        -> blueice3_install : install script, local
        -> db_baks : blueice2 mysql db backups, local
        -> apps
            -> python
                -> BlueIce2 : blueice2 AI core
                    -> checkpoints : Tensorflow checkpoints saves
                    -> data : main app data dir
                        -> access_logs : training, apache access logs
                        -> config : sample config file dir
                        -> evaluate : normal apache access logs to evaluate
                        -> other_vhosts_access : training, apache other vhosts access logs
                        -> output : output directory for writing the bad IP address list
                -> BlueIce2Utils : blueice2utils support app
            -> scripts
                -> SmartServerDefenseCfg : server configuration project (meta)
                    -> blueice3_install : install script, latest
                    -> db_baks : blueice2 mysql db backups, latest
                    -> SmartServerDefenseWeb : documentation and blocked ip listing
            
  2. Apache2: Check that the following packages have been installed.
    apache2

  3. PHP: Check that the following packages have been installed.
    php7.1
    php7.1-common
    libapache2-mod-php7.1
    php7.1-cli
    php7.1-mcrypt
    php7.1-pgsql
    php7.1-curl
    php7.1-mysql

  4. PERL: Check that the following packages have been installed.
    perl

  5. GIT: Check that the following packages have been installed.
    git

  6. MySQL: Check that the following packages have been installed.
    mysql-server
    mysql-client
    php7.1-mysql

  7. GIT Repos (DEPRECATED): Check that the GIT repositories have been initialized. The proper GIT credentials for the dummy account can be found in the install script.
    git clone "https://USER:PASS@github.com/[account]/BlueIce2.git" ./apps/python/BlueIce2
    git clone "https://USER:PASS@github.com/[account]/BlueIce2Utils.git" ./apps/python/BlueIceUtils
    git clone "https://USER:PASS@github.com/[account]/SmartServerDefenseCfg.git" ./apps/scripts/SmartServerDefenseCfg

  8. CURL: Check that the following packages have been installed.
    curl
    libcurl3
    php7.1-curl

  9. MUTT: Check that the following packages have been installed.
    mutt

  10. UNZIP: Check that the following packages have been installed.
    unzip

  11. MLOCATE: Check that the following packages have been installed.
    mlocate

  12. PYTHON: Check that the following packages have been installed.
    python
    python-dev
    python-pip

  13. PYTHON LIBS: Check that the following python libraries have been installed via pip.
    tensorflow
    PyMySql
Checking The MySQL Configuration

App Config Files
The next thing we'll look into is how to double check the base configuration. We'll start with the configuration files that each GIT based application uses. These are created by the install process and should contain the application database account and some environment information depending on the config file in question.
Verify the location of the different config files, there should be a config.txt in each of the directories listed below.
./apps/python/BlueIce2/
./apps/python/BlueIce2Utils/
./apps/scripts/SmartServerDefenseCfg/SmartServerDefenseWeb/

The default config file entries for each application are as follows. The database connection information should match the information contained in the install script. While the settings can be changed they must be consistent between all three projects. The values below are not real running values.

Default Password Formulas...
#!/bin/bash
#Script for generating the default local application database password.
HOST=`hostname`
echo -n "bi2usr_${HOST}" | md5sum | cut -c1-32

#Script for generating the default root database password.
echo -n "root_${HOST}" | md5sum | cut -c1-32

For BlueIce2...
dbconn,dbServer=localhost
dbconn,dbPort=50004
dbconn,dbUser=bi2usr
dbconn,dbPassword=71uTupTu6
dbconn,dbName=blueice2
blueice2utilsexe=../BlueIce2Utils/Main.py
dbValidWebFiles=True
dbTrainingFiles=True
dbLogResult=True

For BlueIce2Utils...
dbconn,dbServer=localhost
dbconn,dbPort=50004
dbconn,dbUser=bi2usr
dbconn,dbPassword=71uTupTu6
dbconn,dbName=blueice2
blueice2exe=../BlueIce2/Main.py

For SmartServerDefenseCfg...
dbconn,dbServer=localhost
dbconn,dbPort=50004
dbconn,dbUser=bi2usr
dbconn,dbPassword=71uTupTu6
dbconn,dbName=blueice2
Python App to Database Connectivity
Next up we'll test our database connections from the different python applications. There is a web site and web service tier with some functionality but we are leaving it out of the initial release to focus on more core aspects of the project.
In order to test the database connection for the BlueIce2Utils application run the following command.
cd ./apps/python/BueIce2Utils
python Main.py -j testDbConn
You should see output like the following. If not you need to double check that the app database account was created successfully. Check that all conig.txt files have the same login credentials and make sure that those credentials work. Recreate the database user account if you have to.
BlueIce2Utils Application Version: 0.4.0.9_python2.7
Found job: testDbConn
Running job: testDbConn
Description: (u'Hello World!', 253, None, 12, 12, 31, False)
Row: Hello World!
Process finished with exit code 0
In order to test the database connection for the BlueIce2 application run the following command.
cd ./apps/python/BlueIce2
python Main.py -dbTestConnOnly
You should see output like the following. If not you need to double check that the app database account was created successfully. Check that all conig.txt files have the same login credentials and make sure that those credentials work. Recreate the database user account if you have to.
Testing database connection...
Description: (u'Hello World!', 253, None, 12, 12, 31, False)
Row: Hello World!
Process finished with exit code 0
Checking The Database Setup
Once we have made sure our app database account is working properly we'll want to check the database is setup correctly by running a series of unit tests on the different database tables. If things don't look good we'll restore the latest db backup from the project directory to our local MySQL server.
cd ./apps/scripts/SmartServerDefenseCfg/db_baks/
mysql -D blueice2 -u bi2usr -p -P 50004 -h localhost < ./unit_tests_apache_log_files_table.sql
mysql -D blueice2 -u bi2usr -p -P 50004 -h localhost < ./unit_tests_blocked_ips_table.sql
mysql -D blueice2 -u bi2usr -p -P 50004 -h localhost < ./unit_tests_training_files_table.sql
mysql -D blueice2 -u bi2usr -p -P 50004 -h localhost < ./unit_tests_valid_web_files_table.sql
Each script should complete without error. We recommend using non-standard ports for the MySQL server and also only allowing localhost connection for the app database account. If you need to restore the database to its default run the following commands.
cd ./apps/scripts/SmartServerDefenseCfg/db_baks/
mysql -D blueice2 -u bi2usr -p -P 50004 -h localhost -e "DROP DATABASE blueice2;"
mysql -D blueice2 -u bi2usr -p -P 50004 -h localhost -e "CREATE DATABASE blueice2;"
mysql -D blueice2 -u bi2usr -p -P 50004 -h localhost < ./blueice2_bak_10-30-2017.sql
Verify the unit test scripts work with the restored database, also verify the python application database connectivity, as detailed above.
Python App Console Arguments

BlueIce2
We'll quickly go over the available command line arguments for BlueIce2. The options are as listed below. There cannot be any spaces between each console argument flag and its value if there is one.
  • -dbServer=[value]: Sets the database server, overriding what has been loaded from the local config.txt if available.
  • -dbPort=[value]: Sets the database port number, overriding what has been loaded from the local config.txt if available.
  • -dbUser=[value]: Sets the database user, overriding what has been loaded from the local config.txt if available.
  • -dbPassword=[value]: Sets the database password, overriding what has been loaded from the local config.txt if available.
  • -dbName=[value]: Sets the database name, overriding what has been loaded from the local config.txt if available.
  • -dataDir=[value]: Sets the location of the data directory, default value is './data'.
  • -apacheLogFile=[value]: Sets the location of the Apache server's access log file, default value is './data/evaluate/apache_eval_log.txt'. This is a copy of the current active log file or a sym link to the current active log file depending on your automation implementation.
  • -webDir=[value]: Sets the location of the web document root, uses this directory for a file system scan when no database link to valid web files is provided.
  • -ipOutputFile=[value]: Sets the location for the IP output file, this file contains the list of IP addresses that should be blocked based on the last, incremental, log scan. The default value for the output file is './data/output/ip_block_list.txt'.
  • -lastLogRow=[value]: Sets the value for the last log row that was read, if the current log size is smaller than the last known log size it is assumed the log has been cycled and this value will be forced to 0.
  • -siteName=[value]: Sets the name of the site used to pull valid web files for that web site. In this way blueice2 lends itself to being run with separate configurations for each web site you want protected. Default value is 'default'.
  • -dbValidWebFiles: Toggles a flag that forces blueice2 to load its valid web file list from the database.
  • -dbTrainingFiles: Toggles a flag that forces blueice2 to load its training file list from the database.
  • -dbTestConnOnly: Toggles a flag that forces blueice2 only run a database connectivity test.
  • -skipEvalCallBack: Toggles a flag that forces blueice2 to skip the eval callback method.
  • -dbLogResult: Toggles a flag that forces blueice2 to record the last read log row of the run in the apache_log_files table. The next run will pick up after this row, unless a log cycle was detected then it will always start at row 0.
  • -logBatchSize=[value]: Sets the batch size of apache log files to process. Example, setting a value of 256 means that the program will wait until 256 logs rows have been read in before processing them with the AI engine. Recommended values for basic servers are 256, 512, 1024.
  • -exeToRun=[value]: Specifies which executable configuration to run. Adding executable configurations is supported but requires adding to the blueice2 python programming.
BlueIce2Utils
We'll quickly go over the available command line arguments for BlueIce2Utils. The options are as listed below.
  • -dbServer=[value]: Sets the database server, overriding what has been loaded from the local config.txt if available.
  • -dbPort=[value]: Sets the database port number, overriding what has been loaded from the local config.txt if available.
  • -dbUser=[value]: Sets the database user, overriding what has been loaded from the local config.txt if available.
  • -dbPassword=[value]: Sets the database password, overriding what has been loaded from the local config.txt if available.
  • -dbName=[value]: Sets the database name, overriding what has been loaded from the local config.txt if available.
  • -neverBlock=[value]: A string with a list of IPs to never block.
  • -ipOutputFile=[value]: Sets the location for the IP output file, this file contains the list of IP addresses that should be blocked based on the last, incremental, log scan.
  • -apacheLogFile=[value]: Sets the location of the Apache server's access log file, default value is './data/evaluate/apache_eval_log.txt'. This is a copy of the current active log file or a sym link to the current active log file depending on your automation implementation.
  • -j scanDir [directory]: Scans the target directory provided recursively and lists the files.
  • -j testDbConn: Tests the current database connection settings.
  • -j clearValidWebFiles: Clears all valid_web_files table entries.
  • -j storeValidWebFiles [directory] [site name]: Scans the target directory for files recursively and then stores them in the valid_web_files table for use by BlueIce2.
  • -j getValidWebFiles: Returns a listing of the valid_web_files table.
  • -j clearTrainingFiles: Clears all training_files table entries.
  • -j storeTrainingFiles [directory]: Scans the target directory for files recursively and then stores them in the training_files table for use by BlueIce2.
  • -j getTrainingFiles: Returns a listing of the training_files table.
  • -j expireBlockedIps: Process the blocked IP expiration dates and deactivates IPs that have been blocked longer than the time interval.
  • -j addBlockedIp [ip_address]: Adds or activates a blocked_ips entry for the given IP address.
  • -j storeBlockedIps [blueice2 IP output file]: Adds or activates the IP address of each entry in the file.
  • -j getBlockedIps: Returns a listing of the blocked_ips table.
  • -j clearBlockedIps: Clears out the blocked_ips table.
  • -j clearApacheLogFiles: Clears out the apache_log_files table.
  • -j storeApacheLogFile [apache log file] [last log row read]: Adds or activates the apache_log_files table entry with the given last log row value.
  • -j getApacheLogFiles: Returns a listing of the apache_log_files table.
  • -j processApacheLogFile [apache log file]: (Broken in 0.4.0.9 use the direct BlueIce2 CLI call) Processes an apache log file and generates a list of IP addresses to block, requires proper cfgs["exeDir"], cfgs["blueice2exe"] configuration settings. See 'Prep App Config Files' documentation here.
  • -j blockIpsUbuntu16 [blueice2 IP output file, system hosts.deny file]: Same as blockIpsUbuntu16Db.
  • -j blockIpsUbuntu16Db [blueice2 IP output file, system hosts.deny file]: Adds a list of IPs from the database to the system host.deny file specified. Currently requires the generated blocked IPs file for future possible usage manipulating IPs.
  • -j blockIpsUbuntu16File [blueice2 IP output file (currently ignored), system hosts.deny file]: Adds a list of IPs from the blocked IPs table to the system host.deny file specified.
Checking The BlueIce2 Configuration

Checking AI Training Files
Next up we'll go over the BlueIce2 configuration to make sure our tensorflow AI app is properly trained and configured. Let's check the training file listing. Run the following commands.
cd ./apps/python/BlueIce2Utils
python Main.py -j getTrainingFiles
You should see a listing of training files that have been stored in the database. The default location for training files is ./apps/python/BlueIce2/data/access_logs/ for regular apache access logs and ./apps/python/BlueIce2/data/other_vhosts_access/ for other vhosts log files. Training files must be marked up in order to be processed by BlueIce2.
Rows the have a preceeding '1 - - ' are considered valid access attempts, rows with e preceeding '0 - - ' are considered attack access attempts. To gaurantee that BlueIce2 retrains itself you must clear all the files contained in the checkpoints directory of the BlueIce2 application.
cd ./apps/python/BlueIce2/checkpoints
rm *
rm *.*
The next time BlueIce2 is ran it will pick up the new list of training files from the database, requires the proper command line arguments, or the local data directory, automatic, and process them. You'll notice that the checkpoints directory will get filled with new files when BlueIce2 runs through new training data.
Adding AI Training Files
If your BlueIce2 installation is new or you would like to add new training files then follow the steps below. Because of the correlation between the access log signature and the intent of the access you can keep the default training files in the access_logs and other_vhosts_access folders.
Place your marked up, '1-- ' or '0-- ' prefixed rows, log files into the access_logs folder if the original log file came from apache's regular access log file. Place your marked up log files in the other_vhosts_access folder if the log file came form apache's other vhosts access log file. The vim command for adding a prefix to each line is listed below this can be used to setup each log file to be prepared as a training file. Please view existing training files as an example.
:%s/^/0-- /
Alternately if the marked up log files are in a location where they can be accessed by BlueIce2 you can store the training files in the database. To store training files in the database use the following commands.
cd ./apps/python/BlueIce2Utils
python Main.py -j storeTrainingFiles [path to training file directory #1]
python Main.py -j storeTrainingFiles [path to training file directory #2]
Remember if you use the database for training files you will need to use the '-dbTrainingFiles' argument when running BlueIce2, otherwise BlueIce2 will automatically load the files in the access_logs and other_vhosts_access directories. To clear the entries in the training_files tables run the following command.
cd ./apps/python/BlueIce2Utils
python Main.py -j clearTrainingFiles
Once you have your training files setup properly either on the file system or in the database you are ready to run BlueIce2. To use the database for training files pass in the -dbTrainingFiles argument otherwise just use the local data directories. Use the -skipEvalCallBack argument to prevent BlueIce2 from evaluating apache log files and generating an output IP list.
Training the AI
In order to help process the marked up apache logs we need to provide our Tensor Flow AI not only with training files but also with a list of valid web files. There are two ways to tell BlueIce2 about valid web files. The first approach passes a web directory argument into BlueIce2 so that it can scan that directory for valid web URLs. The commands for this approach are as follows.
cd ./apps/python/BlueIce2
python Main.py -dbTrainingFiles -skipEvalCallBack -webDir /var/www/html/myWebSite/
To use the database to store our list of valid web files run the following commands. And then use the -dbValidWebFiles argument when running BlueIce2.
cd ./apps/python/BlueIce2Utils
python Main.py -j storeValidWebFiles /var/www/html/myWebSite default_site
To clear the existing valid web files entries use the following command.
cd ./apps/python/BlueIce2Utils
python Main.py -j clearValidWebFiles
To run BlueIce2 and use the database as the source for valid web files use the following commands.
cd ./apps/python/BlueIce2
python Main.py -dbTrainingFiles -skipEvalCallBack -dbValidWebFiles
You'll need to make one adjustment to the small_interval_blueice3 script or run the script creation step again. Make sure you change the exeToRun argument text to "blue_ice_log_reg_nourl" to "blue_ice_log_reg" this will allow the URL to be part of the AI decision making process. The purpose of using the URL in the AI logic is that it can learn to allow valid mistakes, i.e. a valid mistake and not an attack attempt.
Running the AI and Generating a Block List
Once we have our training files in place either via the MySQL BlueIce2 database or the file system and we've ran BlueIce2 to complete the training you should have seen an output similar to the one depicted below. Now we can actually use our trained AI engine to evaluate an active log file.
Found tensor dimension: 7x2
Weight: (7, 2)
Bias: (2,)
('Loss: ', [1.9658128])
Accuracy: 0.9838
Custom Evaluation: blue_ice
Test1: [Bad] False
Test2: [Good] True
Test3: [Bad] False
Test4: [Good] True
Process finished with exit code 0
That's a 98% accuracy after training on a few thousand rows of data. The rating is based on the test set taken from the training set, so we are comparing apples to apples. Strangely enough the included training file won't perform terribly on your own system. But you'll want to include your own training files, clear out all the existing check points, and re-run the training. You'll want to make sure you've included a list of valid web files and directory roots. BlueIce2 uses these when checking the apache log file's URL attribute.
Be sure to keep bBlueIce2 up to date by adding new marked up training files into the database or file system and by updating the valid web files listing when the web site is updated. It is not recommended to use the file system scan approach in a scheduled task. Below is the command for running BlueIce2 and allowing the program to evaluate ./evaluate/apache_eval_log.txt and generate a list of IP addresses to block. For a database approach...
cd ./apps/python/BlueIce2
python Main.py -dbLogResult -dbTrainingFiles -dbValidWebFiles
For a database driven set of training files and a live file system scan for valid web files...
cd ./apps/python/BlueIce2
python Main.py -dbLogResult -dbTrainingFiles -webDir /var/www/html/
For file system driven approach...
cd ./apps/python/BlueIce2
python Main.py -dbLogResult -webDir /var/www/html/
You'll notice the output IP address file has been updated. You should see something like the following list of attacking IP addresses and a row count marker on the last row. The file is located here ./apps/python/BlueIce2/data/output/ip_block_list.txt.
46.17.47.89
115.231.218.25
35.154.189.116
139.162.124.167
189.233.243.115
91.200.12.95
---RowCount:446
Clean BlueIce3
We'll quickly go over the available command line arguments for cleaning out any an instance of BlueIce3 and resetting it for a clean run.
[full path]/blueice3_install -i -int -apkg -j 28
To clear out the TensorFlow checkpoints folder run the following commands. Command #32 is only available in the newer software suite so you may need to run the delete commands by hand if you're using version 0.4.0.9.
[full path]/blueice3_install -i -int -apkg -j 32
or...
cd [full path]/apps/python/BlueIce2
rm ./checkpoints/*
rm ./checkpoints/*.*
Copyright © 2018    Middlemind LLC.    Victor G. Brusca