Bitbucket Pipeline EC2 Deploy

1. **Create an EC2 Key Pair**: Make sure you have an EC2 key pair that can be used to SSH into your EC2 instance. You'll need the private key for this process.

2. **Store the EC2 Private Key in Bitbucket**:
– Go to your Bitbucket repository settings.
– Navigate to **Repository settings > Pipelines > Environment variables**.
– Add a new environment variable for the EC2 private key (e.g., `EC2_PRIVATE_KEY`).

3. **Prepare your EC2 instance**:
– Ensure that your EC2 instance allows SSH access from Bitbucket (typically, you will need to open port 22 for the EC2 instance’s security group).
– Set up a directory on the EC2 instance where the code will be pushed, and make sure it has proper write permissions.

4. **Configure the Bitbucket Pipeline YAML file**:
– Use the following example pipeline configuration:

image: atlassian/default-image:2  # Use a suitable Docker image for your environment

pipelines:
  default:
    - step:
        name: Deploy to EC2
        caches:
          - docker
        script:
          - apt-get update && apt-get install -y openssh-client
          - mkdir -p ~/.ssh
          - echo "$EC2_PRIVATE_KEY" > ~/.ssh/id_rsa  # Use the private key stored in Bitbucket
          - chmod 600 ~/.ssh/id_rsa
          - ssh-keyscan -H <your-ec2-instance-public-ip> >> ~/.ssh/known_hosts  # Add EC2 instance to known hosts
          - rsync -avz --delete ./ <your-ec2-username>@<your-ec2-instance-public-ip>:<destination-folder-on-ec2>

### Breakdown of the steps:
– **Install SSH Client**: The pipeline installs the necessary SSH client (`openssh-client`) to enable communication with EC2.
– **Private Key Setup**: The pipeline creates the SSH key file in the `~/.ssh` directory and sets appropriate permissions.
– **Add EC2 to Known Hosts**: The pipeline ensures that the EC2 instance’s IP is recognized by SSH to avoid SSH authenticity checks.
– **Push Code with Rsync**: The `rsync` command is used to push the code from your Bitbucket repository to the EC2 instance. You can specify the target directory on the EC2 instance with `destination-folder-on-ec2`.

### Things to customize:
– Replace “ with your actual EC2 instance IP address.
– Replace “ with the SSH username for your EC2 instance (e.g., `ec2-user` for Amazon Linux, `ubuntu` for Ubuntu, etc.).
– Replace “ with the path where you want the code to be pushed on the EC2 instance (e.g., `/var/www/html/your-app`).

This setup will ensure that every time you push changes to your Bitbucket repository, the code will be automatically uploaded to the specified folder on the EC2 instance.

### Updated `bitbucket-pipelines.yml`:

image: atlassian/default-image:2  # Use a suitable Docker image for your environment

pipelines:
  default:
    - step:
        name: Deploy to EC2
        caches:
          - docker
        script:
          - apt-get update && apt-get install -y openssh-client
          - mkdir -p ~/.ssh
          - echo "$EC2_PRIVATE_KEY" > ~/.ssh/id_rsa  # Use the private key stored in Bitbucket
          - chmod 600 ~/.ssh/id_rsa
          - ssh-keyscan -H ec2-16-170-254-93.eu-north-1.compute.amazonaws.com >> ~/.ssh/known_hosts  # Add EC2 instance to known hosts
          - rsync -avz --delete ./ ubuntu@ec2-16-170-254-93.eu-north-1.compute.amazonaws.com:/path/to/your/destination-folder-on-ec2

### Key points to update:
– **`/path/to/your/destination-folder-on-ec2`**: Replace this with the actual folder on your EC2 instance where you want the files to be deployed. For example, `/var/www/html/your-app`.
– **`$EC2_PRIVATE_KEY`**: Make sure you have stored your EC2 private key in Bitbucket's environment variables (e.g., `EC2_PRIVATE_KEY`), so it can be used for SSH access.

This pipeline will install the necessary SSH client, set up the private key for SSH access, add the EC2 instance to the list of known hosts to avoid any SSH warnings, and finally use `rsync` to push the code to your EC2 instance.

Let me know if you need further adjustments!

A `deploy.sh` script usually includes the following tasks:

1. **Environment Setup**: Ensuring the correct environment is prepared, which could include setting environment variables or installing dependencies.
2. **Code Sync/Transfer**: Uploading your code to the remote server (using `rsync`, `scp`, or other tools).
3. **Server Configuration**: Running necessary server-side commands, like installing packages, restarting services, or migrating databases.
4. **Post-Deployment Tasks**: Running tests, cleaning up, or notifying the team about the deployment.

### Example `deploy.sh` for your EC2 Deployment:

Since you are deploying to an EC2 instance and you are using a Bitbucket pipeline, you could have a `deploy.sh` script like this:

#!/bin/bash

# Exit on error
set -e

# Print commands and their arguments as they are executed
set -x

# SSH into the EC2 instance (make sure you have the private key set up)
echo "$EC2_PRIVATE_KEY" > ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_rsa

# Add EC2 host to known hosts to avoid SSH authenticity check prompt
ssh-keyscan -H ec2-16-170-254-93.eu-north-1.compute.amazonaws.com >> ~/.ssh/known_hosts

# Sync files to the EC2 instance
rsync -avz --delete ./ ubuntu@ec2-16-170-254-93.eu-north-1.compute.amazonaws.com:/path/to/your/destination-folder-on-ec2

# Optionally, restart a service or run any post-deployment tasks
# Example: Restarting Nginx or Apache (if you are using it)
# ssh -i ~/.ssh/id_rsa ubuntu@ec2-16-170-254-93.eu-north-1.compute.amazonaws.com 'sudo systemctl restart nginx'

# Optionally, clean up or notify the team
# Example: Send a success message to Slack or notify via email

### How to Use `deploy.sh`:
1. **Make it executable**:
Before running the script, ensure it's executable by running:

chmod +x deploy.sh

2. **Run it manually or integrate it in your pipeline**:
– You can execute it manually using `./deploy.sh`.
– Alternatively, you can run this script inside your Bitbucket pipeline by calling it from the `bitbucket-pipelines.yml` file:

script:
       - ./deploy.sh

### Advantages of Using `deploy.sh`:
– **Centralized Automation**: The `deploy.sh` script centralizes deployment logic, making it easier to manage and modify.
– **Consistency**: Running the same deployment process every time ensures consistency across environments (staging, production, etc.).
– **Easier to Debug**: It can be more straightforward to debug a shell script with logging, as you can easily see where a deployment might have failed.

Let me know if you'd like further details on any part of it!

### When to use `deploy.sh`:
– **Consistency**: If you want to ensure that the deployment process is the same across different environments (e.g., development, staging, production), using a script like `deploy.sh` ensures that all the necessary steps are executed the same way each time.
– **Complex Deployments**: If your deployment involves multiple steps (e.g., transferring files, setting environment variables, restarting services, etc.), a script helps organize and automate those tasks, so you don't have to repeat them manually.
– **Ease of Maintenance**: Having a single deployment script makes it easier to maintain and modify the deployment process. If you need to change the way you deploy your code, you only need to update the script.

### When you don't need `deploy.sh`:
– **Simple Deployment**: If your deployment is straightforward, like pushing files to an EC2 instance or simply syncing code, and you don’t need to perform additional actions like restarting services or running post-deployment tasks, you could directly configure those commands in your pipeline (`bitbucket-pipelines.yml`) without a `deploy.sh` script.
– **Minimal Changes**: If you're using a tool (like AWS CodeDeploy, Docker, or any other CI/CD service) that abstracts the deployment logic for you, you might not need a custom script like `deploy.sh`.

### In your case (Bitbucket Pipeline with EC2):
– If you only need to upload files to the EC2 instance and you don’t need to do anything more complex (like restarting services, migrating databases, etc.), you can configure the deployment directly in the Bitbucket pipeline (as shown in my earlier response).
– If your deployment involves several steps (e.g., transferring files, restarting the web server, setting environment variables), you might consider using a `deploy.sh` script to organize these tasks.

### Simplified Bitbucket Pipeline Without `deploy.sh`:
Here’s a simplified version of your pipeline that directly performs the tasks without needing a separate `deploy.sh` script:

image: atlassian/default-image:2  # Use a suitable Docker image for your environment

pipelines:
  default:
    - step:
        name: Deploy to EC2
        caches:
          - docker
        script:
          - apt-get update && apt-get install -y openssh-client
          - mkdir -p ~/.ssh
          - echo "$EC2_PRIVATE_KEY" > ~/.ssh/id_rsa  # Use the private key stored in Bitbucket
          - chmod 600 ~/.ssh/id_rsa
          - ssh-keyscan -H ec2-16-170-254-93.eu-north-1.compute.amazonaws.com >> ~/.ssh/known_hosts  # Add EC2 instance to known hosts
          - rsync -avz --delete ./ ubuntu@ec2-16-170-254-93.eu-north-1.compute.amazonaws.com:/path/to/your/destination-folder-on-ec2

### Conclusion:
– **Not Necessary**: If your deployment is simple, you don’t need a `deploy.sh` script. You can directly configure it in the Bitbucket pipeline.
– **Helpful for Complex Deployments**: If your deployment process involves multiple steps or you need to manage the process more efficiently, then using a `deploy.sh` script can be beneficial.

Ultimately, it depends on your project’s needs and how complex you expect the deployment process to be.

### Solution:

To resolve this, you'll need to ensure that the SSH agent is running before you use `ssh-add`. Here's how you can do that:

1. **Start the SSH agent**:
Before running the `ssh-add` command, you need to start the SSH agent. You can do this by adding the following commands to your script:

eval $(ssh-agent -s)  # Start the SSH agent
   echo "$SSH_PRIVATE_KEY" | tr -d 'r' | ssh-add -  # Add your private key

2. **Ensure the SSH key is properly set up**:
– Ensure that `$SSH_PRIVATE_KEY` is set in your environment, either by passing it as an environment variable in Bitbucket or using Bitbucket’s secure repository variables.
– If you're using Bitbucket Pipelines, ensure you've set the private key correctly in the repository settings under **Repository settings > Pipelines > Environment variables**.

3. **Check the pipeline script**:
Make sure that the key setup part is integrated into your pipeline like this:

image: atlassian/default-image:2  # Choose your Docker image

   pipelines:
     default:
       - step:
           name: Deploy to EC2
           script:
             - apt-get update && apt-get install -y openssh-client
             - eval $(ssh-agent -s)  # Start the SSH agent
             - echo "$SSH_PRIVATE_KEY" | tr -d 'r' | ssh-add -  # Add the SSH private key
             - mkdir -p ~/.ssh
             - ssh-keyscan -H ec2-16-170-254-93.eu-north-1.compute.amazonaws.com >> ~/.ssh/known_hosts  # Add EC2 to known hosts
             - rsync -avz --delete ./ ubuntu@ec2-16-170-254-93.eu-north-1.compute.amazonaws.com:/path/to/your/destination-folder-on-ec2

### Key Points:
– The `ssh-agent` needs to be running for `ssh-add` to work, as it handles the private key for authentication.
– The `eval $(ssh-agent -s)` command starts the agent and allows you to add the private key using `ssh-add`.
– The `echo "$SSH_PRIVATE_KEY" | tr -d 'r' | ssh-add -` command adds the private key to the agent. The `tr -d 'r'` part is used to remove carriage return characters, which might be present if the key was copied from a Windows machine.

### Testing Locally:
If you want to test this locally, ensure you start the SSH agent with:

eval $(ssh-agent -s)

Then add the key manually using:

echo "$SSH_PRIVATE_KEY" | tr -d 'r' | ssh-add -

Let me know if this helps or if you encounter any further issues!

### Possible Causes and Solutions:

1. **SSH Key Format**:
– The private key may be incorrectly formatted or contain extra characters (like carriage returns or line breaks) when copied into the environment variable.

**Solution**:
– Ensure that your private key is properly formatted (without extra characters) before adding it to the environment variable. You can manually clean up the key by opening it in a text editor and making sure there are no extra lines or spaces at the beginning or end.
– If you're using an environment variable to store the private key, you might need to verify that it's correctly stored without any extra characters, especially `r` (carriage returns). You can use `tr -d 'r'` to remove them as you're already doing.

2. **Ensure Correct Permissions**:
– The private key must have the correct permissions. It should only be readable by the user running the command (i.e., no other users should have read access).

**Solution**:
– Set the proper file permissions for the private key:

chmod 600 ~/.ssh/id_rsa

3. **Check for PEM or OpenSSH Key Format**:
– If your private key is in **PEM format** and the `ssh-add` tool doesn't recognize it, you may need to convert it to the OpenSSH format.

**Solution**:
– To convert a PEM key to OpenSSH format, you can use the following command:

ssh-keygen -p -f ~/.ssh/id_rsa -m PEM

4. **Bitbucket Pipelines Specific**:
– If you're using **Bitbucket Pipelines**, ensure that the private key is correctly added in the environment variable and there are no additional newline or carriage return characters. Bitbucket sometimes strips newlines from environment variables, causing issues with key parsing.

**Solution**:
– If you're storing the key in Bitbucket's environment variables, ensure it's correctly formatted:
– Make sure that the key in the Bitbucket environment variable is a **single line**. If you're using multi-line keys, ensure you're handling line breaks properly.

– Use `base64` encoding to store the key and decode it in your pipeline:
1. First, encode your private key in base64 and store it in Bitbucket as an environment variable:

base64 id_rsa > id_rsa.base64

2. Then, in your pipeline, decode it before adding it to the SSH agent:

echo "$SSH_PRIVATE_KEY" | base64 --decode > ~/.ssh/id_rsa
          chmod 600 ~/.ssh/id_rsa
          eval $(ssh-agent -s)
          ssh-add ~/.ssh/id_rsa

5. **Debugging the Key**:
– You can test the key manually by attempting to load it using `ssh-add` on your local machine:

ssh-add -l  # List loaded keys
     ssh-add ~/.ssh/id_rsa  # Add the key

This helps verify if the key itself is valid and can be added correctly to the agent.

### Example Updated Pipeline:
If you're using Bitbucket Pipelines and want to store the key base64 encoded, here’s how you could update your pipeline configuration:

image: atlassian/default-image:2  # Use a suitable Docker image for your environment

pipelines:
  default:
    - step:
        name: Deploy to EC2
        script:
          - apt-get update && apt-get install -y openssh-client
          - eval $(ssh-agent -s)  # Start the SSH agent
          - echo "$SSH_PRIVATE_KEY" | base64 --decode > ~/.ssh/id_rsa  # Decode and add the SSH private key
          - chmod 600 ~/.ssh/id_rsa
          - ssh-keyscan -H ec2-16-170-254-93.eu-north-1.compute.amazonaws.com >> ~/.ssh/known_hosts  # Add EC2 to known hosts
          - rsync -avz --delete ./ ubuntu@ec2-16-170-254-93.eu-north-1.compute.amazonaws.com:/path/to/your/destination-folder-on-ec2

### Summary:
– Ensure your private key is in the correct format (PEM or OpenSSH).
– Make sure to handle any carriage returns or line breaks properly, especially when storing the key in Bitbucket environment variables.
– You might want to base64 encode the key for safer transmission and decoding in the pipeline.

Let me know if you need further assistance!

pipelines:
default:
– step:
name: Deploy Code to EC2
caches:
– node
script:
– apt-get update && apt-get install -y openssh-client
– eval $(ssh-agent -s) # Start the SSH agent
– echo "$SSH_PRIVATE_KEY" | tr -d 'r' | ssh-add –
– mkdir -p ~/.ssh
– chmod 700 ~/.ssh
– echo "$EC2_SSH_PUBLIC_KEY" > ~/.ssh/authorized_keys
– chmod 600 ~/.ssh/authorized_keys
– rsync -avz –exclude=.git –exclude=node_modules ./ ubuntu@ec2-16-170-254-93.eu-north-1.compute.amazonaws.com:/var/www/shareyourthoughts
– ssh -o StrictHostKeyChecking=no ubuntu@ec2-16-170-254-93.eu-north-1.compute.amazonaws.com "cd /var/www/shareyourthoughts && git pull origin main"

### Key Points to Ensure:
1. **Ensure SSH Agent is Set Up Properly**:
You’ve correctly initialized the SSH agent with `eval $(ssh-agent -s)`, but ensure that the `$SSH_PRIVATE_KEY` environment variable in Bitbucket contains the private key without any extra line breaks or spaces.

2. **Avoid SSH Warning (StrictHostKeyChecking)**:
You've used `-o StrictHostKeyChecking=no`, which is good to avoid SSH connection warnings. This is useful for automating the deployment in a CI/CD pipeline.

3. **Set Permissions Properly**:
You're creating and setting permissions for the `~/.ssh` folder and `authorized_keys`. This ensures that the authorized keys can be used for SSH authentication.

4. **Rsync Command**:
The `rsync` command is used to sync files between your local machine and the EC2 instance, excluding `.git` and `node_modules`, which is generally a good practice.

5. **Git Pull**:
After transferring the files, you're using `git pull` to update the code on the EC2 instance, which should work as long as the necessary git configuration (e.g., SSH keys or HTTPS) is set up on the EC2 instance.

### Optimized Version of Your Pipeline:

Here’s an updated and slightly optimized version of your pipeline configuration:

image: node:21

pipelines:
  default:
    - step:
        name: Deploy Code to EC2
        caches:
          - node
        script:
          - apt-get update && apt-get install -y openssh-client
          - eval $(ssh-agent -s)  # Start the SSH agent
          - echo "$SSH_PRIVATE_KEY" | tr -d 'r' | ssh-add -  # Add the private key to the agent
          - mkdir -p ~/.ssh
          - chmod 700 ~/.ssh
          - echo "$EC2_SSH_PUBLIC_KEY" > ~/.ssh/authorized_keys  # Add public key to authorized keys
          - chmod 600 ~/.ssh/authorized_keys
          - ssh-keyscan -H ec2-16-170-254-93.eu-north-1.compute.amazonaws.com >> ~/.ssh/known_hosts  # Avoid host verification prompts
          - rsync -avz --exclude=.git --exclude=node_modules ./ ubuntu@ec2-16-170-254-93.eu-north-1.compute.amazonaws.com:/var/www/shareyourthoughts
          - ssh -o StrictHostKeyChecking=no ubuntu@ec2-16-170-254-93.eu-north-1.compute.amazonaws.com "cd /var/www/shareyourthoughts && git pull origin main"

### Explanation of Changes:

1. **`ssh-keyscan` for Known Hosts**:
I've added the `ssh-keyscan` command to automatically add the EC2 instance to the list of known hosts, which prevents the SSH client from asking whether to trust the host (this is essential for automation).

2. **Private Key Handling**:
You're already handling the private key correctly by using `tr -d 'r'` to strip any carriage return characters, which is useful if you're using a key copied from Windows systems.

3. **No Need for `chmod 700 ~/.ssh` (if not needed)**:
The `chmod 700 ~/.ssh` is not always strictly required unless you're going to add other SSH keys into this folder. If it’s part of your best practices, it's fine to keep, but it’s usually not necessary if you’re only working with the `authorized_keys` file.

4. **Environment Variables**:
Make sure that:
– `$SSH_PRIVATE_KEY` is correctly set as an environment variable in Bitbucket (contains the private key for SSH authentication).
– `$EC2_SSH_PUBLIC_KEY` is the corresponding public key added to `authorized_keys` on your EC2 instance.

### Additional Recommendations:
– **Testing SSH Access**: If you run into issues where the SSH connection is not working, you can add debugging steps such as `ssh -v` to get verbose output:

ssh -v -o StrictHostKeyChecking=no ubuntu@ec2-16-170-254-93.eu-north-1.compute.amazonaws.com

This will provide more information about the connection and why it might be failing.

– **Git Configuration on EC2**: Make sure that the EC2 instance has the necessary SSH key (the corresponding public key) to pull from the Git repository, especially if you're using SSH for Git authentication. Alternatively, you could use HTTPS with a personal access token if you encounter any issues with SSH.

### Conclusion:
This setup should automate your deployment smoothly. If everything is configured correctly, this pipeline will push your code to the EC2 instance and perform a `git pull` to update the code on the instance. Let me know if you run into any further issues!

1. **Incorrect Key Format**: The SSH private key may not be in a valid OpenSSH format or may have been altered when copying or saving it.

2. **Corrupted or Invalid Key**: If the private key is corrupted (e.g., extra characters, newlines, etc.), it will fail to load.

### Steps to Resolve the Issue:

#### 1. **Verify the Private Key Format**
Ensure that your SSH private key is in a valid OpenSSH format. The key should look like this:

-----BEGIN OPENSSH PRIVATE KEY-----
...
-----END OPENSSH PRIVATE KEY-----

If your key is in a different format (like PEM), it might need to be converted.

#### 2. **Check for Extra Line Breaks or Carriage Returns**
Sometimes when copying and pasting the private key, extra line breaks or carriage return characters (`r`) may be added, which could cause issues. You are already using `tr -d 'r'` to remove carriage returns, but it's possible there might be extra characters or newlines that still affect it.

Try to manually clean up the key, ensuring no extra spaces, newlines, or invalid characters exist before storing it in your environment variable.

#### 3. **Reformat the Key (if needed)**
If the key is not in the OpenSSH format, you can convert it to the OpenSSH format using the following command on your local machine:

ssh-keygen -p -f /path/to/your/id_rsa -m PEM

If the private key is already in PEM format, the command above will convert it to OpenSSH format.

Alternatively, if the key is in a different format (like `PEM`), and you want to convert it to `OpenSSH`, use this:

ssh-keygen -p -f /path/to/your/id_rsa -m PEM

#### 4. **Use Base64 Encoding for Key in Bitbucket Pipelines**
To safely store and pass the private key in environment variables in Bitbucket Pipelines, it is often a good idea to encode the key using `base64` and decode it during the pipeline.

Here’s how you can encode the private key:

1. **Encode the private key** on your local machine:

base64 ~/.ssh/id_rsa > id_rsa.base64

2. **Store the base64-encoded key** in Bitbucket's environment variables:
– In your Bitbucket repository, go to **Repository settings > Pipelines > Environment variables** and set the `SSH_PRIVATE_KEY` variable with the base64-encoded key.

3. **Modify your pipeline to decode the key**:
In your Bitbucket Pipeline, decode the key before using it with `ssh-add`:

image: node:21

   pipelines:
     default:
       - step:
           name: Deploy Code to EC2
           caches:
             - node
           script:
             - apt-get update && apt-get install -y openssh-client
             - eval $(ssh-agent -s)  # Start the SSH agent
             - echo "$SSH_PRIVATE_KEY" | base64 --decode > ~/.ssh/id_rsa  # Decode and store the SSH private key
             - chmod 600 ~/.ssh/id_rsa  # Set correct permissions for the private key
             - ssh-add ~/.ssh/id_rsa  # Add the private key to the agent
             - mkdir -p ~/.ssh
             - chmod 700 ~/.ssh
             - echo "$EC2_SSH_PUBLIC_KEY" > ~/.ssh/authorized_keys  # Add the public key to authorized_keys
             - chmod 600 ~/.ssh/authorized_keys
             - ssh-keyscan -H ec2-16-170-254-93.eu-north-1.compute.amazonaws.com >> ~/.ssh/known_hosts  # Add EC2 to known hosts
             - rsync -avz --exclude=.git --exclude=node_modules ./ ubuntu@ec2-16-170-254-93.eu-north-1.compute.amazonaws.com:/var/www/shareyourthoughts
             - ssh -o StrictHostKeyChecking=no ubuntu@ec2-16-170-254-93.eu-north-1.compute.amazonaws.com "cd /var/www/shareyourthoughts && git pull origin main"

This approach avoids potential issues with line breaks or incorrect key formats by encoding and decoding the key.

#### 5. **Manual Testing**
You can also manually test if the SSH key is valid and can be added by running the following commands on your local machine:

ssh-add -l  # List all currently loaded keys
ssh-add ~/.ssh/id_rsa  # Add the private key to the SSH agent

This will help identify if the key is valid on your local machine before proceeding with Bitbucket Pipelines.

### Summary:
– Ensure the key is in OpenSSH format.
– Check for extra line breaks or carriage returns.
– Use `base64` encoding for safer handling of the key in Bitbucket Pipelines.
– Ensure the correct permissions for the SSH private key (use `chmod 600 ~/.ssh/id_rsa`).

Let me know if you need further assistance!