About java : Why-does-my-log-file-say-server-failed-to-start-but-application-is-running

Would like to preface I am by no means an AWS/Linux/Shell expert, some of my setup is probably not the greatest but most of it has been working for me so far.

I am running a Spring Boot app on an AWS EC2 instance (AWS Linux). I have setup CodeDeploy to deploy my source code, build it, and run the produced jar on the EC2 instance when I push to my main GitHub branch.

The app starts up just fine and runs. However, the log file that I am writing to always states:

APPLICATION FAILED TO START

description: Web Server failed to start. Port 80 was already in use

The app is running just fine on port 80 and is accessible via my domain I have setup to point to my EC2 instance’s elastic ip.

When connected to the instance I have run ps -ef | grep jarName and found that two results come back. One is sudo nohup java -jar jarName.jar and the other is java -jar jarName.jar, am I somehow running the jar twice and that is why it is “failing” in the log file? When I stop everything and manually run sudo java -jar jarName.jar it runs fine, when I close it and look at the log file everything looks fine.

A few files that may be of interest:

• clean.sh (runs BeforeInstallation in appspec.yml):

cd /home/ec2-user
echo "killing jarName"
#Force to kill all "jarName" processes
SERVICES=$(ps -ef | grep jarName | grep -v grep | awk '{print $2}')
if [ $SERVICES != "" ]; then
    echo "killing jarName services already running"
    sudo kill -9 $SERVICES
fi

if [ "$?" != "0" ]; then 
    echo "Killing process failed"
    exit 1
fi
# delete the app folder if it already exists
echo "removing sourceFolder folder"
sudo rm -rf sourceFolder
if [ "$?" != 0 ]; then
    echo "Removing folder failed"
    exit 1
fi

• gradleBuild.sh (runs as AfterInstallation in appspec.yml):

# runs the gradle build command
echo "starting gradleBuild.sh"
cd /home/ec2-user/sourceFolder
sudo ./gradlew --stop
sudo ./gradlew clean -x test build
if [ "$?" != "0" ]; then 
    echo "error building jar"
    exit 1
fi 

• startServer.sh (runs as ApplicationStart in appspec.yml):

echo "starting server"
cd /home/ec2-user/sourceFolder/path/jarLivesHere
# run as background and also close things according to AWS docs
sudo java -jar ./jarName.jar > /dev/null 2> /dev/null < /dev/null & 

• Lastly my logback configuration file:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>

    <property name="LOGS" value="./logs" />
    <timestamp key="startTimeStamp" datePattern="yyyy-MM-dd'_'HH-mm-ss.SSS"/>
    <appender name="Console"
        class="ch.qos.logback.core.ConsoleAppender">
        <layout class="ch.qos.logback.classic.PatternLayout">
            <Pattern>
                %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n
            </Pattern>
        </layout>
    </appender>

    <appender name="RollingFile"
        class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${LOGS}/spring-boot-logger-${startTimeStamp}.log</file>
        <encoder
            class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <Pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</Pattern>
        </encoder>

        <rollingPolicy
            class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <!-- rollover daily and when the file reaches 10 MegaBytes -->
            <fileNamePattern>${LOGS}/archived/spring-boot-logger-%d{yyyy-MM-dd}.%i.log
            </fileNamePattern>
            <timeBasedFileNamingAndTriggeringPolicy
                class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
                <maxFileSize>10MB</maxFileSize>
            </timeBasedFileNamingAndTriggeringPolicy>
        </rollingPolicy>
    </appender>
    
    <!-- LOG everything at INFO level -->
    <root level="info">
        <appender-ref ref="RollingFile" />
        <appender-ref ref="Console" />
    </root>

</configuration>

Appreciate any and all input/ideas!

I believe I have somehow solved this, or it just started to magically work more consistently. What I did:

1. In my build.gradle, I disabled a “plain” jar that was being generated (a “fat” jar and plain jar were both being created). Not sure if this was really part of the problem
2. I updated my scripts to include more echo statements to try and gather more information. This showed that occasionally I was having issues with killing the already running processes/services. In my first if statement of clean.sh I changed $SERVICES to "$SERVICES"
3. I ran a search for the services again in startServer.sh to verify if anything was running immediately before calling to start the jar

After doing those 3 things, it seemed to start more consistently and the log file is being updated as expected.

If I find more out in the future I will try to update this response also.