Deploy a Python Flask Project on AWS EC2 (high level)

Create instance

1. save key and mod
2. Modify IAM Role

3. create Elastic IPs

4. update ssh config

nano ~/.ssh/config

Host ec2winjob_ca
    HostName ec2-xxx-117.us-west-2.compute.amazonaws.com
    User ec2-user
    IdentityFile /Users/xxx/key_pair_aws_ec2_win_.pem
    IdentitiesOnly yes
    AddKeysToAgent yes
    UseKeychain yes

Set up python enviorment

after login by ssh:

1. install python

sudo yum update -y
sudo yum install -y python3-pip python3-devel

2. create file folders:

sudo mkdir -p /var/www/winjob_us
sudo mkdir -p /var/www/logs
sudo mkdir -p /var/www/temp
sudo mkdir -p /var/www/rollback

3. change ownership

sudo chown -R ec2-user:ec2-user /var/www
sudo chmod -R 755 /var/www

4. change the permissions

To ensure that any new files or directories created within /var/www also inherit these permissions, you can set default ACLs.

sudo setfacl -Rdm u:ec2-user:rwx /var/www
sudo setfacl -Rm u:ec2-user:rwx /var/www

After setting the ACLs, verify that they are applied correctly using getfacl:

getfacl /var/www

5. upload python files

logout ssh, using local terminal:

rsync -av --exclude='.env' \
        --exclude='.git/' \
        --exclude='.github/' \
        --exclude='.DS_Store' \
        --exclude='venv_winjob' \
        --exclude='flask_session' \
        --exclude='zzz_backup' \
        --exclude='test.py' \
        --exclude='logs_local' \
        --exclude='tool' \
        --exclude='temp' \
        --exclude='__pycache__' \
        --exclude='app/blueprints/__pycache__/' \
        --exclude='app/utils/__pycache__/' \
        --exclude='.gitattributes' \
        --exclude='.gitignore' \
 -e "ssh -i /Users/henrywu/MyDrive/98_Products/config/key_pair_aws_ec2_winjob_us.pem" \
 /Users/henrywu/MyDrive/99_Coding/01-Github/winjob_ai/ \
 ec2-user@ec2-34-214-237-225.us-west-2.compute.amazonaws.com:/var/www/winjob_us

6. create environment

log in ssh, and:

python3 -m venv /var/www/winjob_us/venv_winjob
source /var/www/winjob_us/venv_winjob/bin/activate

pip3 install pip-tools
pip3 install -r /var/www/winjob_us/requirements.txt

Set up Gunicorn

1. Install Gunicorn

Gunicorn is a WSGI HTTP Server for UNIX, used to run Python web applications. It acts as an interface between your web application written in Python and the web. It can serve your application directly to the internet, although it’s typically not designed to directly handle high loads, static files, SSL, and other concerns in production environments.

Ensure Gunicorn Uses the Virtual Environment. You should install Gunicorn inside your virtual environment to ensure it uses the correct Python interpreter and can access the Flask installation and other dependencies located there.

source /var/www/winjob_us/venv_winjob/bin/activate
sudo chown -R ec2-user:ec2-user /var/www/winjob_us/venv_winjob

pip3 install gunicorn

After installing Gunicorn in your virtual environment, check again with which gunicorn to ensure it points to the one in your virtual environment.

which gunicorn

2. test

cd /var/www/winjob_us
gunicorn --workers 3 --bind 0.0.0.0:8000 run:app

Remember to kill the pid after test:

First, find out which process is currently using port 8000. You can do this using the lsof or netstat command. If lsof is installed, use:

sudo lsof -i:8000

sudo kill -9 <PID>

Replace <PID> with the actual Process ID of the process using port 8000. The -9 option sends a SIGKILL signal, which forcibly stops the process.

3. Create a Gunicorn systemd Service File

Create a new service file for Gunicorn in /etc/systemd/system/, e.g., gunicorn.service.

sudo nano /etc/systemd/system/gunicorn.service

Add the Following Configuration: Adjust the paths and settings according to your application setup.

[Unit]
Description=gunicorn daemon
After=network.target

[Service]
User=ec2-user
Group=nginx
WorkingDirectory=/var/www/winjob_us
ExecStart=/var/www/winjob_us/venv_winjob/bin/gunicorn --workers 3 --bind 0.0.0.0:8000 run:app

[Install]
WantedBy=multi-user.target

4. create group

sudo groupadd nginx
sudo usermod -a -G nginx ec2-user

5. Start and Enable the Gunicorn Service:

sudo systemctl daemon-reload
sudo systemctl start gunicorn
sudo systemctl enable gunicorn

6. Check Status:

To make sure Gunicorn is running, you can use:

sudo systemctl status gunicorn

Set up Nginx

Nginx is a web server that can also be used as a reverse proxy, load balancer, mail proxy, and HTTP cache. In the context of web applications, it’s often used in front of application servers like Gunicorn to serve static files, handle HTTPS, manage load balancing, and provide additional buffering for the application servers.

1. A web server like Nginx or Apache serves as a reverse proxy to handle HTTP requests and forward them to your Flask application.

sudo yum install nginx

sudo chown -R nginx:nginx /var/www/winjob_us

2. Update the server part:

sudo nano /etc/nginx/conf.d/winjob_us.conf

server {
    listen 80;
    listen  [::]:80;
    server_name 34.214.237.XXX;

    location / {
        proxy_pass http://localhost:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

3. Start the web server and set it to launch on boot.

sudo systemctl start nginx
sudo systemctl enable nginx
sudo systemctl status nginx

If everything goes alright, you can visit the website by the public IP address of your EC2 instance even you log out SSH:

Set up SSL

First, ensure that your domain correctly point to your server's IP address where Nginx is running.

install Certbot, which automates the process of obtaining and renewing Let’s Encrypt certificates:

sudo dnf update -y
sudo dnf install -y certbot python3-certbot-nginx

sudo certbot --nginx -d winjob.us -d www.winjob.us

If there is an issue:

update config files:

sudo nano /etc/nginx/conf.d/winjob_us.conf

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name winjob.us www.winjob.us;

    ssl_certificate /etc/letsencrypt/live/winjob.us/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/winjob.us/privkey.pem;
    ssl_session_cache shared:SSL:1m;
    ssl_session_timeout 10m;
    ssl_ciphers HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers on;

    location / {
        proxy_pass http://localhost:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location = /sitemap.xml {
        root /usr/share/nginx/html;
    }

    location = /robots.txt {
        root /usr/share/nginx/html;
    }

}

server {
    listen 80;
    listen [::]:80;
    server_name winjob.us www.winjob.us;

    location / {
        return 301 https://$host$request_uri;
    }
}

Check for syntax errors:

sudo nginx -t

Reload Nginx to apply changes

sudo systemctl reload nginx

If you prefer to have Certbot attempt the installation again:

sudo certbot install --cert-name winjob.us

Once you’ve configured everything and reloaded Nginx, you can test your SSL setup using:

• Browser: Visit https://winjob.us and https://www.winjob.us to ensure the SSL certificate is active and the site is accessible via HTTPS.
• SSL Test: Use tools like SSL Labs’ SSL Test to check your server’s SSL configuration and certificate validity.

Install Redis on AWS EC2

Amazon Linux typically does not come with Redis in its default repository, so you might need to enable additional repositories or install Redis manually.

sudo yum install redis6 -y

After installation, you need to start the Redis service. You can do this using the following command:

sudo systemctl start redis6
sudo systemctl enable redis6
sudo systemctl status redis6

Update yml file:

name: Deploy to AWS EC2

on:
  push

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout repository
      uses: actions/checkout@v2

    - name: Setup SSH and Transfer Files
      run: |
        # Setup SSH Key
        mkdir -p ~/.ssh
        echo "${{ secrets.SSH_PRIVATE_KEY_WINJOB_US }}" > ~/.ssh/id_rsa
        chmod 600 ~/.ssh/id_rsa
        ssh-keyscan -H ${{ secrets.EC2_HOST_WINJOB_US }} >> ~/.ssh/known_hosts

        # Generate timestamp and backup folder path on the runner
        TIMESTAMP=$(TZ="America/Vancouver" date +"%Y%m%d_%H%M")
        BACKUP_FOLDER="/var/www/rollback/$TIMESTAMP"

        # Rsync the project to the backup folder on EC2
        rsync -av --exclude='.env' \
        --exclude='.git/' \
        --exclude='.github/' \
        --exclude='.DS_Store' \
        --exclude='venv_winjob' \
        --exclude='flask_session' \
        --exclude='zzz_backup' \
        --exclude='test.py' \
        --exclude='logs_local' \
        --exclude='tool' \
        --exclude='temp' \
        --exclude='__pycache__' \
        --exclude='app/blueprints/__pycache__/' \
        --exclude='app/utils/__pycache__/' \
        --exclude='.gitattributes' \
        --exclude='.gitignore' \
        -e "ssh -i ~/.ssh/id_rsa" \
        $GITHUB_WORKSPACE/ ec2-user@${{ secrets.EC2_HOST_WINJOB_US }}:"$BACKUP_FOLDER/"

        # Execute commands on the EC2 instance
        ssh -i ~/.ssh/id_rsa ec2-user@${{ secrets.EC2_HOST_WINJOB_US }} <<EOF
        # Update the footer with the timestamp
        sed -i "s|<!--TIMESTAMP-->|Last Deployed: ${TIMESTAMP}|g" "$BACKUP_FOLDER/app/templates/base_chat_webpage.html"

        # Prepare the deployment directory
        sudo chown -R ec2-user:ec2-user /var/www/winjob_us/
        sudo rm -rf /var/www/winjob_us/*
        sudo cp -r $BACKUP_FOLDER/. /var/www/winjob_us/
        EOF

    - name: Post-deployment Commands
      run: |
        ssh -i ~/.ssh/id_rsa ec2-user@${{ secrets.EC2_HOST_WINJOB_US }} << 'EOF'

        # Create the virtual environment if it doesn't exist
        python3 -m venv /var/www/winjob_us/venv_winjob

        # Activate the virtual environment
        source /var/www/winjob_us/venv_winjob/bin/activate
        
        # Install the required packages
        pip3 install -r /var/www/winjob_us/requirements.txt
        pip3 install gunicorn
        
        # Restart your application
        sudo systemctl restart gunicorn
        EOF

Github parameters

Back Up

Creating a snapshot of an Amazon EC2 instance is a common method for backing up your instance’s volumes on AWS. A snapshot captures the state of an Elastic Block Store (EBS) volume at a specific point in time, which can be used later to restore or create duplicates of the volume. Here’s how to create a snapshot and use it for backup purposes:

1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
2. In the navigation pane, click on Elastic Block Store -> Volumes.
3. Select the volume you want to snapshot.
4. Click Actions -> Create Snapshot.
5. Provide a description for the snapshot so you can identify it later, and click Create Snapshot.