Shell Scripting for Daily AWS Resource Monitoring

What is a Shell and Why is Shell Scripting Needed?

A shell is a program that allows users to communicate with the operating system by typing commands. It acts as an interface between the user and the system. Shell scripting is the process of writing a series of commands in a script to automate tasks, making it more efficient and consistent.

Write a Shell Script to Report the Usage of AWS in Your Project

In this example, we need to generate a daily report on the usage of AWS resources such as EC2 instances, AMIs, S3 buckets, and Lambda functions. The report must be provided every day at 6 PM to the manager. By creating a shell script, we can automatically collect the data, saving time and reducing errors.

Setting Up AWS CLI on EC2

For this, we can refer to your previous blog aws-cli-and-roles about AWS CLI.

Steps to Create the Shell Script

To create the script, we can use the Vim editor.

vim aws_resources_tracker.sh

We can refer to the official CLI documentation to learn and understand the AWS CLI commands.https://docs.aws.amazon.com/cli/latest/reference/

#!/bin/bash
########################
#Author: Anju
# Date: 4/8/2025
# #  Version: v1
# #  This report will show the AWS usages
# #########################

# # AWS EC2
# # AWS IAM
# # AWS S3
# # AWS Lambda
# #########################
# Use full paths for commands
#
OUTPUT_FILE=/home/ubuntu/ResourceTracker #Full path for output file
AWS=/usr/local/bin/aws   # Path to AWS CLI
JQ=/usr/bin/jq           # Path to jq

# ResourceTracker: we will redirect the output to this file
set -x #to run this command in dbug mode

#List ec2 instances
echo "Print list of ec2 instances" > $OUTPUT_FILE
$AWS ec2 describe-instances | $JQ '.Reservations[].Instances[].InstanceId' >> $OUTPUT_FILE
echo "" >> $OUTPUT_FILE #this line is added to add some space between 2 outputs

#List s3 buckets
echo "Print list of S3 buckets" >> $OUTPUT_FILE
$AWS s3 ls >> $OUTPUT_FILE
echo "" >> $OUTPUT_FILE

#List iam users
echo "Print list of iam users" >> $OUTPUT_FILE
$AWS iam list-users >> $OUTPUT_FILE
echo "" >> $OUTPUT_FILE

# List lambda functions
echo "Print list of lambda functions" >> $OUTPUT_FILE
$AWS lambda list-functions >> $OUTPUT_FILE
echo "" >> $OUTPUT_FILE

Why Specify Full Paths for AWS and jq in the Script?

When running the script in a cron job, the environment is minimal and doesn't automatically have the same PATH as interactive shell. By specifying the full paths for commands like aws and jq, ensure that the cron job can find and run these commands properly.

OUTPUT_FILE=/home/ubuntu/ResourceTracker #Full path for output file
AWS=/usr/local/bin/aws   # Path to AWS CLI
JQ=/usr/bin/jq           # Path to jq

This ensures that script works in the cron environment as expected, even if the cron's default PATH doesn’t include the directories where these tools are located.

can use the following commands to find the path of aws and jq:

which aws
which jq

Crontab: To integrate the script with crontab, need to understand how it works.

• Ensure the Script is Executable: Make sure the script is executable. If not, use this command to make it executable:

    chmod 700 /home/ubuntu/aws_resources_tracker.sh
  

• Edit the Crontab

  To schedule the script using cron, need to add an entry in crontab. Here's how we can do that:

  1. Open crontab for editing:

    crontab -e

This will open the crontab file in your default text editor (e.g., vi, nano).

2. Add an entry to schedule script at 5:43 Am everyday. A cron job follows this format along with path of script:

    43 5 * * * /home/ubuntu/aws_resources_tracker.sh

The five asterisks represent the following time fields (in order):

    pgsqlCopy* * * * *  user-name command-to-execute
    - - - - -
    | | | | |
    | | | | +---- Day of the week (0 - 6) (Sunday=0)
    | | | +------ Month (1 - 12)
    | | +-------- Day of the month (1 - 31)
    | +---------- Hour (0 - 23)
    +------------ Minute (0 - 59)

You can also check if the output file ResourceTracker is being updated as expected based on the schedule you chose.

5. Consider Logging Output (Optional)

like to capture the output and any potential errors (e.g., if something fails), can modify the crontab entry like so:

    43 5 * * * /home/ubuntu/aws_resources_tracker.sh >> /path/to/logfile.log 2>&1

This will append both standard output and errors to logfile.log.

Example Final Crontab Entry:

If script is located at /home/ubuntu/aws_resources_tracker.sh, and you want it to run daily at 5:43 AM, your crontab entry would be:

    43 5 * * * /home/ubuntu/aws_resources_tracker.sh >> /home/ubuntu/aws_resource_report.log 2>&1

• 43 5 → Runs every day at 5:43 AM

• * * * → Any day, any month, any weekday

• /home/ubuntu/aws_resources_tracker.sh → Runs your script

• >> /home/ubuntu/aws_resource_report.log 2>&1 → Overwrites the file each time with the script’s output

• 2>&1 → Includes any error messages along with regular output

our script is now scheduled to run at the specified times via cron.

Challenges

Here, we will discuss the challenges we faced while writing this script and automating it.

• #!/bin/bash: When starting the script, this shebang is used to specify that the script should be executed with the Bash shell only.

• Cron jobs run with a minimal environment and don't have access to your usual PATH, so specifying the full paths to commands like aws and jq ensures they are found and executed correctly. It guarantees your script works reliably regardless of the environment.

• When specifying the time for cron, make sure that the timezone you are using matches the timezone of your server.