Smart Server Defense
Powered by BlueIce3
Installation Documentation

This page has extra information on the steps that are run during the base installation process. This information may be useful to those trying to setup the software on a different flavor of linux or to those who want to know more about the project and how it works.
Install Script Console Arguments
Below is a listing of the CLI arguments supported by the install script.
  • Option '-i': is the installation flag. This will run through all the instalation steps and is considered an action.
  • Option '-u': is the update flag. This will run through the daily update steps and is considered an action.
  • Option '-dbb': is the database backup flag and will run a simple MySQL dump of the blueice2 database.
  • Option '-int': is the interactive mode OFF flag. Interactive mode is set to on by default. In interactive mode you will be prompted before each installation or update step. Turn this flag off if you want the installation to proceed without user input - this is not recommended unless you are an advanced user.
  • Option '-j #': allows the user to specify running a specific job for an action command, as indicated by the value that replaces the # sign. So, for instance, if you ran the install script with the following arguments '-u -j 1 -int' the install script would run the daily update, only job #1, and would not prompt the user for input before running any steps.
  • Option '-apkg': is a flag that will make all package installation via apt-get automated and not prompt the user for input.
  • Option '-h': is a flag that will display the help information for the script.
  • Option '-dbcmd': is a flag that will print MySQL database CLI command syntax. This is a convenience call for quickly setting up the commands needed for quick console MySQL calls.
  • Option '-q': is a flag that will cause the script to immediately exit.
  • Option '-sshdc': is a flag that will cause a check on the current SSHD port number, it will change the current SSHD port number to match that specified in the script and restart the SSH daemon.
Default Steps
During each run of the install script, whether it is for installation or daily updates and maintenance, performs a few checks on the system.
  • Hostname: Check if AWS hostname is properly set in the hosts file. Sometimes this adjustment is needed on fresh AWS Ubuntu instances.
  • SSHD Port (Deprecated): Force the SSHD port to a non-standard port number. You can find this port number at the top of the install script. This adjustment is for security purposes and to prevent daily attacks on standard SSH ports. Don't forget to adjust your security-group if you are using AWS. Only run by default if SSHD_CHECK_ENABLED is set to true at the script header.
Update Steps
When run in update mode (-u) the install script will automatically update, if the '-int' argument has been provided, the linux packages, blueice2 software and file system db for the given server. The automation setup documentation has more information on the details of using the update steps as an automated daily server update. Below are the steps run during the update process. You can run the entire update process like so, ./blueice3_install -u -int, or each command individually like so, ./blueice3_install -u -j #.
  1. Update Ubuntu: Runs the following package manager commands in Ubuntu.
    sudo apt-get update -y
    sudo apt-get upgrade -y
    sudo apt-get dist-upgrade -y
    sudo apt-get auto-remove -y

  2. Run GIT Update (DEPRECATED): Runs the git pull and git status commands on the following local code repositories listed below.
    ./apps/python/BlueIce2: BlueIce2 AI software.
    ./apps/python/BlueIceUtils: Automation tools for BlueIce2.
    ./apps/scripts/SmartServerDefenseCfg: Documentation, web site for BlueIce2.

  3. Update File System DB: Runs the following command to update the server's file system db.
    sudo updatedb
Install Steps
When run in install mode (-i) the installation script will install, it is not recommended to use the '-int' argument during this process, and configure the linux packages and blueice2 software necessary. The configuration setup documentation has more information on the details of the installation. Below are the steps run during the install process. You can run the entire update process like so, ./blueice3_install -i, or each command individually like so, ./blueice3_install -i -j #. Keep in mind these install steps are not only for Smart Server Defense, but also for setting up a capable bare bones system.
  1. Create Directories: Checks and/or creates the necessary directories in the root directory of the install script.
    ./apps/python/BlueIce2: BlueIce2 AI software.
    ./db_baks: Repository for database backups.
    ./apps/python/BlueIceUtils: Automation tools for BlueIce2.
    ./apps/scripts/SmartServerDefenseCfg: Documentation, web site for BlueIce2.

  2. Install APACHE: Installs the Apache 2 web server from the package manager.
    sudo apt-get install -y apache2

  3. Install PHP, PERL: Installs PHP 7 and PERL from the package manager.
    sudo apt-get install -y php7.1
    sudo apt-get install -y php7.1-common libapache2-mod-php7.1 php7.1-cli php7.1-mcrypt php7.1-pgsql php7.1-curl php7.1-mysql
    sudo apt-get install -y perl

  4. Install GIT: Installs and configures GIT. Git is currently used to keep the blueice2 code repositories up to date.
    sudo apt-get install -y git
    git config --global user.email 'EMAIL'
    git config --global user.name 'USERNAME'

  5. Install MySQL: Installs MySQL server and client software necessary to manage the blueice2 database as well as for install script interaction database backups and restores.
    sudo apt-get install -y mysql-server mysql-client
    sudo apt-get install -y php7.1-mysql

  6. Prep MySQL(DEPRECATED): Runs the mysql_secure_installation script. This step may require more user interaction then other steps.
    /usr/bin/mysql_secure_installation

  7. Prep GIT (DEPRECATED): Preps the GIT installation by running git clone commands on the local code repositories. You can check the install script for the directory paths, and repository URLs.
    git clone "https://USER:PASS@github.com/vbrusca/BlueIce2.git" ./apps/python/BlueIce2
    git clone "https://USER:PASS@github.com/vbrusca/BlueIce2Utils.git" ./apps/python/BlueIceUtils
    git clone "https://USER:PASS@github.com/vbrusca/SmartServerDefenseCfg.git" ./apps/scripts/SmartServerDefenseCfg

  8. Update GIT (DEPRECATED): Runs the git pull commands on the following local code repositories listed below.
    ./apps/python/BlueIce2: BlueIce2 AI software.
    ./apps/python/BlueIceUtils: Automation tools for BlueIce2.
    ./apps/scripts/SmartServerDefenseCfg: Documentation, web site for BlueIce2.

  9. Install CURL: Runs the CURL package install commands listed below.
    sudo apt-get install -y curl
    sudo apt-get install -y libcurl3 php7.1-curl

  10. Install MUTT: Runs the MUTT package install commands listed below for CLI email support, just in case, this is an optional install.
    sudo apt-get install -y mutt

  11. Install UNZIP: Runs the UNZIP package install commands listed below for CLI unzip support, just in case, this is an optional install.
    sudo apt-get install -y unzip

  12. Install MLOCATE: Runs the MLOCATE package install commands listed below for CLI mlocate support.
    sudo apt-get install -y mlocate
    sudo updatedb

  13. Install JAVA JDK: Runs the JAVA JDK package install commands listed below, this is an optional install.
    sudo apt-get install -y default-jdk

  14. Install PYTHON 2.7: Runs the PYTHON 2.7 package install commands listed below.
    sudo apt-get install -y python python-pip python-dev

  15. Restart APACHE: Restarts the currently running APACHE web server.
    sudo service apache2 restart

  16. Restart MySQL: Restarts the currently running MYSQL database server.
    sudo service mysql restart

  17. Update MLOCATE: Updates the mlocate file system database.
    sudo updatedb

  18. Auto Start MySQL: Updates the system configuration to automatically start MySQL on boot.
    sudo update-rc.d mysql defaults

  19. Auto Start APACHE: Updates the system configuration to automatically start APACHE on boot.
    sudo update-rc.d apache2 enable

  20. Update APACHE Conf: Updates the APACHE web server configuration so that the log format entries match the format expected by blueice2. The install script uses sed commands to check and replace the existing log format entries. The correct, apache default, log format entries are listed below.
    APACHE_LOG_PATTERN_1="LogFormat \"%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"\" vhost_combined"
    APACHE_LOG_PATTERN_2="LogFormat \"%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"\" combined"
    APACHE_LOG_PATTERN_3="LogFormat \"%h %l %u %t \"%r\" %>s %O\" common"
    APACHE_LOG_PATTERN_4="LogFormat \"%{Referer}i -> %U\" referer"
    APACHE_LOG_PATTERN_5="LogFormat \"%{User-agent}i\" agent"

  21. Update MySQL Conf: Updates the MySQL database server configuration so that the default port is a non-standard port. The install script uses sed commands to check and replace the existing default port number for MySQL. The port number matching strings are listed below.
    MYSQL_PORT_STR="port=3306"
    MYSQL_PORT_NUMBER_STR_NEW="port=50004"
    MYSQL_PORT="50004"

  22. Update BASH Profile: Updates the current user's BASH profile to include a shortcut for running the server update scripts. The install script uses sed to make the adjustment to the user's profile but the bash command is listed below. The WORKING_DIR is the directory that the blueice3_install script has been installed to.
    BASH_PROFILE_STR="alias updatebi=\"WORKING_DIR/blueice3_install -u -int\""

  23. Install PYTHON LIBS: Installs Tensor Flow and other required python libraries via pip.
    sudo pip install tensorflow
    sudo pip install PyMySql

  24. Install MySQL User: Installs an application level MySQL user for MySQL access via blueice2 and blueice2utils. The user account is created using CLI MySQL tools. You can check the values of the script variables in the actual install script.
    mysql -u root -p -P ${MYSQL_PORT} "GRANT ALL PRIVILEGES ON '${MYSQL_DB}' TO '${MYSQL_DEVUSER}'@'localhost' IDENTIFIED BY '${MYSQL_DEVPASS}';"

  25. Restore MySQL DB: Restores the latest backup of the MySQL database for this version of the software. The restore is performed using CLI MySQL tools. You can check the values of the script variables in the actual install script.
    mysql -u root -p -P ${MYSQL_PORT} ${MYSQL_DB} < ${MYSQL_RESTORE_FILE}

  26. Prep App Config Files: This step will create a configuration text file in the directory of each software repository. This will enable all the support software to connect to the database and share a proper set of configuration files. Config files, config.txt, are located in the following directories. Please check the main install script for the current application database user credentials or set your own.
    ./apps/python/BlueIce2/
    ./apps/python/BlueIce2Utils/
    ./apps/scripts/SmartServerDefenseCfg/SmartServerDefenseWeb/

    The default config file entries for each application are as follows. The values below are not real running values.

    Default Password Formulas...
    #!/bin/bash
    #Script for generating the default local application database password.
    HOST=`hostname`
    echo -n "bi2usr_${HOST}" | md5sum | cut -c1-32

    #Script for generating the default root database password.
    echo -n "root_${HOST}" | md5sum | cut -c1-32

    For BlueIce2...
    dbconn,dbServer=localhost
    dbconn,dbPort=3306
    dbconn,dbUser=bi2usr
    dbconn,dbPassword=71uTupTu6
    dbconn,dbName=blueice2
    blueice2utilsexe=../BlueIce2Utils/Main.py
    dbValidWebFiles=True
    dbTrainingFiles=True
    dbLogResult=True

    For BlueIce2Utils...
    dbconn,dbServer=localhost
    dbconn,dbPort=3306
    dbconn,dbUser=bi2usr
    dbconn,dbPassword=71uTupTu6
    dbconn,dbName=blueice2
    blueice2exe=../BlueIce2/Main.py

    For SmartServerDefenseCfg...
    dbconn,dbServer=localhost
    dbconn,dbPort=3306
    dbconn,dbUser=bi2usr
    dbconn,dbPassword=71uTupTu6
    dbconn,dbName=blueice2

  27. Add CRONTAB Entries: Generates the short term and long term scripts needed for crontab automation in the local directory. Adds the necessary crontab entries if they are missing.
    #BLUEICE3 SCHEDULED TASKS
    */5 * * * * root [full path]/small_interval_blueice3
    5 23 * * * root [full path]/large_interval_blueice3

  28. Clean BlueIce3: Cleans the BlueIce3 instalation and resets the database entries.
    cd [full path]/BlueIce2Utils
    python ./Main.py -j cleanValidWebFiles default
    python ./Main.py -j cleanValidWebFiles test_site
    python ./Main.py -j storeValidWebFiles /var/www/html/ default
    python ./Main.py -j cleanTrainingFiles
    python ./Main.py -j storeTrainingFiles [full path]/BlueIce2/data/access_logs/
    python ./Main.py -j storeTrainingFiles [full path]/BlueIce2/data/other_vhosts_access/
    cd [full path]


  29. SSHD Port Check (Deprecated): Ensures the SSHD is set to the proper listening port.
    Uses sed and grep to check the SSHD config file for a port entry matching the values in the header of the script.

  30. Reset Default Passwords: Restores the default password formulas for root and bi2usr accounts. You will need to run command 26, config file generation, after you run this step if the passwords have been changed and the application config files need to be updated.
    1. Brings down the mysql service.
    2. Writes a temporary SQL file to update the two accounts.
    3. Starts the mysql service with a special init file that has been allowed by apparmor.
    4. The mysql service is started back up and the new passwords should have been set.
    5. Cleans up any SQL at the end and leaves a marker file behind that let's the system know the default formula driven password has been set. This last step is for use with AMI images to let the system know it doens't have to prep any formula driven database passwords.

    #!/bin/bash
    #Script for generating the default local application database password.
    $HOST=`hostname`
    echo -n 'bi2usr_${HOST}' | md5sum | cut -c1-32

    #Script for generating the default root database password.
    echo -n 'root_${HOST}' | md5sum | cut -c1-32

Copyright © 2018    Middlemind LLC.    Victor G. Brusca