Http Url and Wildfly HeartBeat

For my projects I need a script that check for http response correctly with 200 or 301 and then check for wildfly process status.

So I write the following script.

The script write in a log file the various steps.
The mail dir is a directory where other daemon check for new file and send it to mail..
The lock files avoid to write too many file on mail dir in case of heart down. It rewrite the error at most each hour.
The script include two url heartbeat, you can copy & paste other ones.
If wildfly process is down the script try to restart.

I'm not a bash script expert feel free to comment in order to correct or improve the scripts!

#!/bin/bash
LOG_FILE=/home/you/url_heartbeat.log #FILE FOR LOGGING
MAIL_DIR_FILE=/home/mailDir/heartbeat.log #FILE PATH FOR COPING THE LOG IF TEST IS NOT PASSED (IN THIS DIR A DAEMON WILL SEND A MAIL)
WILDFLY_SERVER_LOG=/opt/wildfly/standalone/log/server.log #WILDFLY SERVER.LOG TO SEE IF PARTICULAR ERROR OCCURS
CURRENT_HOUR_LOCK_FILE="$(date +%H).fail"
PREVIOUS_HOUR_LOCK_FILE="$(expr $(date +%H) - 1).fail"
rm -f $PREVIOUS_HOUR_LOCK_FILE #REMOVE PREVIOUS HOUR FILE

#DO TEST IF CURRENT_HOUR_LOCK_FILE DOESN'T EXIST
if [ -f $CURRENT_HOUR_LOCK_FILE ];
then
   echo "$(date) Skip Test file $CURRENT_HOUR_LOCK_FILE exist" >> $LOG_FILE
else
  
echo -e  "$(date) start url_heartbeat" > $LOG_FILE
TEST_PASSED=true

RESPONSE_CODE=$(curl -sL -w "%{http_code}" "http://www.example1.com/index.jsp" -o /dev/null) #GET URL RESPONSE CODE

if  [ "$RESPONSE_CODE" = 301 ] || [ "$RESPONSE_CODE" = 200 ]  #IF 301 REDIRECT OR 200 OK 
then
               echo -e  "OK Example1: $RESPONSE_CODE" >> $LOG_FILE
            else
               echo -e  "FAIL Example1: $RESPONSE_CODE" >> $LOG_FILE
               TEST_PASSED=false
            fi
            
RESPONSE_CODE=$(curl -sL -w "%{http_code}" "http://example2/index.jsp" -o /dev/null)

if  [ "$RESPONSE_CODE" = 301 ] || [ "$RESPONSE_CODE" = 200 ] 
then
               echo -e  "OK Example2: $RESPONSE_CODE" >> $LOG_FILE
            else
               echo -e  "FAIL Example2: $RESPONSE_CODE" >> $LOG_FILE
               TEST_PASSED=false
            fi

            
WILDFLY_STATUS=$(/etc/init.d/wildfly status) #CHECK WILDFLY PROCESS STATUS
echo -e  $WILDFLY_STATUS

if  [[ "$WILDFLY_STATUS" == *running* ]]
then
               echo -e  "\nOK Wildfly Status: $WILDFLY_STATUS\n" >> $LOG_FILE
            else
               echo -e  "\nFAIL Wildfly Status: $WILDFLY_STATUS\n" >> $LOG_FILE
               TEST_PASSED=false #IF WILDFLY DOWN TRY TO RESTART
               /etc/init.d/wildfly restart #RESTART COMMAND
               sleep 10m #WAIT RESTARTING
               WILDFLY_STATUS=$(/etc/init.d/wildfly status) #CHECK WILDFLY PROCESS STATUS AGAIN
         echo -e  "\nAfter Restart Wildfly Status: $WILDFLY_STATUS\n" >> $LOG_FILE
         if  [[ "$WILDFLY_STATUS" == *running* ]]
         then
         echo -e "Restart Successfully\n" >> $LOG_FILE
         rm -f $CURRENT_HOUR_LOCK_FILE #TEST FAILED BUT RESTART SUCCESSFULLY SO DELETE LOCK FILE IN ORDER TO DOESN'T SKIP THE TESTS AT NEXT CRON HIT
         fi
            fi

            
echo -e  "test passed: $TEST_PASSED" >> $LOG_FILE
echo -e  "test passed: $TEST_PASSED"
if  [ "$TEST_PASSED" = false ]
then
echo -e  "NOT PASSED\n\n\n $(tail -100 $WILDFLY_SERVER_LOG)" >> $LOG_FILE
touch $CURRENT_HOUR_LOCK_FILE #MAKE A FILE IF TEST IS NOT PASSED WITH CURRENT HOUR AS NAME
cp $LOG_FILE $MAIL_DIR_FILE
            fi
fi

#TEST CASE
#CURRENT_HOUR_LOCK_FILE EXIST -> SKIP TEST
#CURRENT_HOUR_LOCK_FILE NOT EXIST -> DO TEST

#ALL TEST OK -> CHECK LOG FILE FOR ALL OK ECHOES
#AT LEAST ONE RESPONSE_CODE URL TEST NOT OK -> 
#1) SEE THE CURRENT_HOUR_LOCK 
#2) THE LOG_FILE IS COPIED IN MAIL_DIR PATH 
#3) THE NEXT CRON HIT SKIP
#WILDFLY IS NOT RUNNING -> AS THE PREVIOUS CASE AND TRY TO RESTART
#IF RESTART IS OK THE CURRENT_HOUR_LOCK IS DELETED AND NEXT CRON HIT DO THE TEST
#IF RESTART GOES WRONG THE NEXT TEST IS SKIPPED

SSH won't Work - expecting SSH2_MSG_KEX_DH_GEX_GROUP

Short Post for a little-big Problem.

If your firewall is open for ssh, telnet on ssh port works but ssh won't work.

Try to debug the ssh connection by using verbose debug

ssh -v user@yourip

if ssh hangs on

"expecting SSH2_MSG_KEX_DH_GEX_GROUP"

and then ssh returns with the following error: "Read from socket failed: Operation timed out"

Probably you have an MTU/fragmentation problem and you will solve the problem by set correctly a new mtu value for network interface.

So launch by terminal the following command:

sudo ifconfig en1 mtu 576

Where en1 is your active network interface.

Wildfly SSL Client Authentication

In order to ask for a client ssl authentication you have to setup standalone.xml, web.xml and jboss-web.xml in different section.


1) First of all say to UNDERTOW -> https listener that ssl authentication could occurs

<subsystem xmlns="urn:jboss:domain:undertow:1.1">
           <buffer-cache name="default"/>
           <server name="default-server">
               <http-listener name="default" socket-binding="http"/>
               <https-listener name="https-listener" socket-binding="https" security-realm="MySecurityRealm"/>
               <host name="default-host" alias="localhost">
                   <location name="/" handler="welcome-content"/>
                   <access-log pattern="common" directory="${jboss.server.log.dir}" prefix="access"/>
                   <filter-ref name="server-header"/>
                   <filter-ref name="x-powered-by-header"/>
               </host>
           </server>
           <server name="default-server_secondary">
               <http-listener name="default_secondary" socket-binding="http_secondary"/>
               <https-listener name="https-listener_secondary" socket-binding="https_secondary" security-realm="MySecurityRealm_secondary" verify-client="REQUESTED"/>
               <host name="fatturaelettronica01.com" alias="testfoo.com">
                   <location name="/" handler="welcome-content"/>
                   <access-log pattern="common" directory="${jboss.server.log.dir}" prefix="access"/>
                   <filter-ref name="server-header"/>
                   <filter-ref name="x-powered-by-header"/>
               </host>
           </server>
           <servlet-container name="default" default-encoding="utf-8">
               <jsp-config/>
           </servlet-container>
           <handlers>
               <file name="welcome-content" path="${jboss.home.dir}/welcome-content"/>
           </handlers>
           <filters>
               <response-header name="server-header" header-name="Server" header-value="WildFly/8"/>
               <response-header name="x-powered-by-header" header-name="X-Powered-By" header-value="Undertow/1"/>
           </filters>
       </subsystem>

2) Set also in the https-listener security realm that ssl authentication can occurs.

So add <authentication> in order to setup the truststore path (the <ssl> section instead indicate the keystore for server certificate).

 <security-realm name="MySecurityRealm_secondary">
               <server-identities>
                   <ssl protocol="TLSv1">
                       <keystore path="sdi_keystore.jks" relative-to="jboss.server.config.dir" keystore-password="changeit" alias="fatturaelettronica01.com" key-password="changeit"/>
                   </ssl>
               </server-identities>
               <authentication>
                   <truststore path="sdi_cacerts.jks" relative-to="jboss.server.config.dir" keystore-password="changeit"/>
                   <local default-user="$local"/>
                   <properties path="mgmt-users.properties" relative-to="jboss.server.config.dir"/>
               </authentication>
           </security-realm>


3) In order to link the war application with wildfly configuration you have to define a security-domain using login module (certificateRoles login module)

<security-domain name="sdi_webservice_client_cert_domain" cache-type="default">
                   <authentication>
                       <login-module code="CertificateRoles" flag="required">
                           <module-option name="verifier" value="org.jboss.security.auth.certs.AnyCertVerifier"/>
                           <module-option name="securityDomain" value="sdi_webservice_client_cert_domain"/>
                           <module-option name="rolesProperties" value="file:${jboss.server.config.dir}/app-roles.properties"/>
                       </login-module>
                   </authentication>
                   <jsse keystore-password="changeit" keystore-url="file:${jboss.server.config.dir}/sdi_keystore.jks" truststore-password="changeit" truststore-url="file:${jboss.server.config.dir}/sdi_cacerts.jks" client-auth="true"/>
               </security-domain>


Make also the app-roles.properities file with the format
certificate DN=nome_ruolo

# sample app-roles.properties file
CN\=SDI-MRRGPP80L31D960U,\ OU\=Servizi\ Telematici,\ O\=Agenzia\ delle\ Entrate,\ C\=IT=SDI_USER

You have to escape the = symbol and space with backslash like the example.

4) In the web.xml war setup the client-cert authentication type, the role, and set as realm name the security domain name

<?xml version="1.0" encoding="UTF-8"?>
<web-app version="3.1" xmlns="http://xmlns.jcp.org/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee http://xmlns.jcp.org/xml/ns/javaee/web-app_3_1.xsd">
   <session-config>
       <session-timeout>
           30
       </session-timeout>
   </session-config>
   <security-constraint>
       <display-name>Constraint1</display-name>
       <web-resource-collection>
           <web-resource-name>sdi</web-resource-name>
           <description/>
           <url-pattern>/*</url-pattern>
       </web-resource-collection>
       <auth-constraint>
           <description/>
           <role-name>SDI_USER</role-name>
       </auth-constraint>
   </security-constraint>
   <login-config>
       <auth-method>CLIENT-CERT</auth-method>
       <realm-name>sdi_webservice_client_cert_domain</realm-name>
   </login-config>
   <security-role>
       <description/>
       <role-name>SDI_USER</role-name>
   </security-role>
</web-app>

5) Finally also the jboss-web.xml have to set the security domain used

<jboss-web>
   <server-instance>default-server_secondary</server-instance>      
   <virtual-host>fatturaelettronica01.com</virtual-host>
   <security-domain>sdi_webservice_client_cert_domain</security-domain>
</jboss-web>

Wildfly Deploy on different server and host

If you setup different <server> in undertow subsystem and each <server> has its own virtual host then when deploy a war you have to setup the jboss-web.xml correctly


<jboss-web>
   <server-instance>default-server_secondary</server-instance> <!-- does mapping to server -->
   <virtual-host>server.mydomain.com</virtual-host> <!-- does mapping to host inside server -->
</jboss-web>

Wildfly Multiple IP and Multiple SSL Certificate

Wildfly can manage different ip in the same standalone server.
The final target is two manage different ssl certificate for each ip.

Due to ssl handshake limitation each hostname needs to be associate with different ip


For example


https://localhost:8181 send certificate1
https://192.168.1.3:8181 send certificate 2

But first start to setup the new ip interface


<interfaces>
       <interface name="management">
           <inet-address value="${jboss.bind.address.management:127.0.0.1}"/>
       </interface>
       <interface name="public">
           <inet-address value="${jboss.bind.address:127.0.0.1}"/>
       </interface>
       <interface name="public_secondary">
           <inet-address value="${jboss.bind.address:192.168.1.3}"/>
       </interface>
       <interface name="unsecure">
           <inet-address value="${jboss.bind.address.unsecure:127.0.0.1}"/>
       </interface>
   </interfaces>


Adding the public_secondary


Next add to socket bindings section a new socketbinding using the new interface

<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
       <socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>
       <socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9993}"/>
       <socket-binding name="ajp" port="${jboss.ajp.port:8009}"/>
       <socket-binding name="http" port="${jboss.http.port:8080}"/>
       <socket-binding name="https" port="${jboss.https.port:8181}"/>
       <socket-binding name="http_secondary" interface="public_secondary" port="${jboss.http.port:8080}"/>
       <socket-binding name="https_secondary" interface="public_secondary" port="${jboss.https.port:8181}"/>
       <socket-binding name="jacorb" interface="unsecure" port="3528"/>
       <socket-binding name="jacorb-ssl" interface="unsecure" port="3529"/>
       <socket-binding name="messaging-group" port="0" multicast-address="${jboss.messaging.group.address:231.7.7.7}" multicast-port="${jboss.messaging.group.port:9876}"/>
       <socket-binding name="txn-recovery-environment" port="4712"/>
       <socket-binding name="txn-status-manager" port="4713"/>
       <outbound-socket-binding name="mail-smtp">
           <remote-destination host="localhost" port="25"/>
       </outbound-socket-binding>
   </socket-binding-group>

the new socket bindings will be used in undertow configuration in <server> section


<subsystem xmlns="urn:jboss:domain:undertow:1.1">
           <buffer-cache name="default"/>
           <server name="default-server">
               <http-listener name="default" socket-binding="http"/>
               <https-listener name="https-listener" socket-binding="https" security-realm="MySecurityRealm"/>
               <host name="default-host" alias="localhost">
                   <location name="/" handler="welcome-content"/>
                   <access-log pattern="common" directory="${jboss.server.log.dir}" prefix="access"/>
                   <filter-ref name="server-header"/>
                   <filter-ref name="x-powered-by-header"/>
               </host>
           </server>
           <server name="secondary-server">
               <http-listener name="default_secondary" socket-binding="http_secondary"/>
               <https-listener name="https-listener_secondary" socket-binding="https_secondary" security-realm="MySecurityRealm_secondary"/>
               <host name="default-host_secondary" alias="localhost">
                   <location name="/" handler="welcome-content"/>
                   <access-log pattern="common" directory="${jboss.server.log.dir}" prefix="access"/>
                   <filter-ref name="server-header"/>
                   <filter-ref name="x-powered-by-header"/>
               </host>
           </server>
           <servlet-container name="default" default-encoding="utf-8">
               <jsp-config/>
           </servlet-container>
           <handlers>
               <file name="welcome-content" path="${jboss.home.dir}/welcome-content"/>
           </handlers>
           <filters>
               <response-header name="server-header" header-name="Server" header-value="WildFly/8"/>
               <response-header name="x-powered-by-header" header-name="X-Powered-By" header-value="Undertow/1"/>
           </filters>
       </subsystem>


We maked the secondary-server that references the new socket binding.


In order to send a different ssl certificate for https-listener_secondary we assign it a new security realm: MySecurityRealm_secondary


We have to setup 2 security realm one for https-listener and other one for https-listener_secondary

Each realm references same keystore but different certificate alias

<security-realm name="MySecurityRealm">
               <server-identities>
                   <ssl protocol="TLSv1">
                       <keystore path="foo.keystore" relative-to="jboss.server.config.dir" keystore-password="secret" alias="localhost" key-password="secret"/>
                   </ssl>
               </server-identities>
           </security-realm>
            <security-realm name="MySecurityRealm_secondary">
               <server-identities>
                   <ssl protocol="TLSv1">
                       <keystore path="foo.keystore" relative-to="jboss.server.config.dir" keystore-password="secret" alias="testfoo" key-password="secret"/>
                   </ssl>
               </server-identities>
           </security-realm>
       </security-realms>