Category: AWS
Sending Email on Servers on AWS (SES)
Sending email from PHP is often automatic and simple if setup on a traditional hosting account – however the mail() function in PHP does not work on AWS EC2 servers. AWS requires that you use the SES service. (Simple Email Service).
There are several tutorials that walk you through getting your account setup
To send email FROM your email domains you will need to setup and authorize the domain (See TIP #1 below)
https://docs.aws.amazon.com/ses/latest/DeveloperGuide/verify-domains.html
The step that is often missed is that outgoing email in new accounts is allowed ONLY to the email addresses you verify – to verify individual emails addresses.
https://docs.aws.amazon.com/ses/latest/DeveloperGuide/verify-email-addresses.html
To enable your account to send email from your verified domains, and to ANY email address, you will need to get the account removed from the sandbox mode (check the Email Sending -> Sending Statistics page for a message) Click the ‘Request a Sending Limit Increase’ button to ask to remove the account from the sandbox. (here is a tutorial –
https://docs.aws.amazon.com/ses/latest/DeveloperGuide/request-production-access.html )
TIP #1: I would suggest using route 53 for DNS for your domains, as it is a simple couple button clicks to authorize the domains, and get DKIM setup
TIP #2: make sure that you have posted a privacy policy or an email policy on your site – which explains how you send emails and handle their email information. Include a link to your policy in your request for removal from the sandbox.
AWS – Change your Root Password
It is not a good idea to use the Root account to manage and work with your AWS account. Ideally you have setup IAM user accounts with only the required permissions.
However occasionally you need to update your Root level account password. This video quickly shows you how to do it
(if this video is low quality, try opening it full screen and playing it again, the video is short and may be done playing before YouTube can catch up with downloading a higher quality version)
AWS – Disable IAM User access to Billing Console
In an AWS Account, Root users can create IAM Users with Account Administrator Permissions
However those users do not have access to the Billing Reporting.
A Root user can enable this though – follow the steps in this video
(if this video is low quality, try opening it full screen and playing it again, the video is short and may be done playing before YouTube can catch up with downloading a higher quality version)
AWS – Enable IAM User access to Billing Console
In an AWS Account, Root users can create IAM Users with Account Administrator Permissions
However those users do not have access to the Billing Reporting.
A Root user can enable this though – follow the steps in this video
(if this video is low quality, try opening it full screen and playing it again, the video is short and may be done playing before YouTube can catch up with downloading a higher quality version)
AWS CodeDeploy Solutions
We have worked with several clients, to come up with Code Deployment solutions.
In some cases, the solutions is small, on a simple website redesign where the client simply wants to have the revisions history that comes with a Code repository such as GIT or Subversion (SVN).
However in more robust environments, where the client needs to have separate development, test and productions servers (dev, test and live), we often end up building a more involved software development lifecycle (SDLC).
One of the Software Development Life Cycles we use makes use of the AWS platform and the Code Deployment tools. This article is to describe a little bit about how we have used the code deployment tools, and to give Round Up all of the articles we have written as of 2/26/2016 about solving issues with AWS Code Deploy.
CodeDeploy on AWS
CodeDeploy on AWS is a tool which automatically connects to a group of EC2 servers, to deploy code and runs scripts. This automates the labor intensive task of connecting to and consistently updating several each time that a new version of a new website design or development.
The concept behind the way we have used it is. Setup a CodeDeployment group and associate it with an AutoScaling group. When your group grows, the code is deployed to it, When your code is deployed all members of the group receive it.
- make your source code changes in your development environment
- zip up your source code and upload it to an S3 bucket
- create a new revision within your deployment group and start the deployment
The AWS system will, over the course of 1 to 30 minutes, go through each of your servers (depeing on your deployment configuration, I always use OneAtATime)
The following Roundup, lists each of the articles matraex has written about the AWS Code Deployment
Challenges when working with CodeDeploy and an AWS AutoScaling Group
I write about how in one setup I used a GIT repo to store my code, then I write some automated scripts (deploy.php) to check in the code, and deploy it to a test environment, and then to deploy the code to live (each has their own separate group).
In this AWS Auto Scaling Group – CodeDeploy Challenges article
I talk:
- about how errors in the codedeploy-agent are hard to debug
- if a server is terminating while you deploy, the deployment could fail
- in an update, I describe a script command that can be run to clean up the install to prevent some deployment failure
Debugging CodeDeploy
I write about some of the difficulties of understanding what is happening during a code deployment and how to debug it.
The quick of it is to update ‘/etc/codedeploy-agent/conf/codedeployagent.yml’ to have ‘verbose=true’ and restart the codedeployagent and then read the ‘/var/log/aws/codedeploy-agent/codedeployment-agent.log’ file to figure out what happened.
Read the entire article Debugging CodeDeployment on AWS here for more of a description with the actual commands highlighted and for a couple error examples
AWS Auto Scaling Group – CodeDeploy Challenges
AWS Auto Scaling Group – CodeDeploy Challenges
First here is my setup
- A single development / test server in the AWS cloud, backed by a separate Git Repository.
- WHen code is completed in the development environment it is commited to the development branch (using whichever branching scheme best fits the project)
- At the same time the code is merged to the test branch, and the code is available for client testing on the ‘test_stage’ site if they would like
- Then on an as needed basis the code in in the test branch (on the test_stage server) is deployed to AWS using their CodeDeploy api
- git archive test -> deploy.zip
- upload the file an S3 bucket (s3cmd)
- register the zip file as a revision using the AWS Register Revision API call
- This creates a file that can be deployed to any deployment group
- I setup two groups in my AWS account , test and live.
- When the client is ready, I run a script which deploys thes the ziped up revision to the Test server, where they are able to look atit and approve.
- Then I use the same method but move it instead of the www deployment group.
(The complexities of setting this up are deeper than I am going in this article, but for future prospects, all of this programming knowledges is stored in our deploy.php file)
A couple of tricks “they” dont tell you.
- Errors can be difficult to debug – if you update your code deployment to do more verbose logging it can help you to determine what some of the errors were.
- update /etc/codedeploy-agent/conf/codedeployment.yml, set verbose to yes.
- restart the service /etc/init.d/code-deployment restart (it can take several minutes to restart, this is normal)
- tail the log files to watch a deployment in real time, or investigate it after the fact (tail /var/log/aws/codedeploy-agent)
- Deploying a Revision to servers while they may be going through some termination instability, may likely cause your deployment to fail when one of you servers terminates.
- To prevent this, update the deployment autoscaling plan to have a minim and a maximum of the server, and do not take it under load during the 10 – 15 minutes (up to 2 hours) issues will cause errors
- Depending on the load on your servers, your deployment could take a lot of cpu and could generate an autoscaling alert and could spin up new tasks or send you an email. There is not a correct way to deal with this, however it is a good idea to know about it before you deploy.
- Finally the item that I wrote this because of, it appears that when you attempt to deploy a revision to an autoscaling group, it can cause some failures.
- The obvious one is that the deployment will fail if it is attempted while the server is shutting down
- However, it seems that if you have decided to upgrade your AMI, and your Launch Configuration, that a deployment will fail. And for me, it actually caused a key failure to login as well (this could have been because of multiple server terminations and then another server took over the IPs within a few minutes) Anyway, much caution about these things.
UPDATE:
Well, the problem was actually that the by ‘afterinstall.sh’ script, was cleaning up the /opt/codedeployment/ directory (so we didn’t run out of space after a couple dozen deployments), but I was also removing the appspec.yml file.
So I updated the command that runs in the afterinstall to be
/usr/bin/find /opt/codedeploy-agent/deployment-root/ -mindepth 2 -mtime +1 -not -path '*deployment-instruction*' -delete
Debugging CodeDeployment on AWS
Debugging CodeDeployment on AWS
This article is being written well after I have already installed the CodeDeployment daemon on an ubuntu server, created an AMI out of it and set it up as an auto launch server from a Scaling Group.
I am documenting the process I went through to dig into the error a bit more, this helps to identify and remember where the logs files are and how to get additional information, even if the issue is never the same again.
Issues with running the codedeployment showed up as a python error in the log
more /var/log/aws/codedeploy-agent/codedeployment-agent.log put_host_command_complete(command_status:"Failed",diagnostics:{format:"JSON",payload:"{"error_code":5,"script_name":"","message":"not opened for reading","log":""}"
I decided this is not enough information to troubleshoot an error so I had to dig in and fina way to make it more verbose. I found this file, just update verbose from false to true
vi /etc/codedeploy-agent/conf/codedeployagent.yml
Then restart the codedeploy-agent
/etc/init.d/code-deployagent restart
This can take quite a while since it runs quite a bit of background installing and checking for duplicate processes. but once it is complete you can check that the process is running again.
ps ax|grep codedeploy
Once this is running in verbose mode, monitor the log
tail -f /var/log/aws/codedeploy-agent/codedeployment-agent.log
and re-run the deployment and view the results of the log, the most useful thing for me was to grep for the folder that more specific error information was written to.
grep -i "Creating deployment" /var/log/aws/codedeploy-agent/codedeployment-agent.log
This showed me the folder that all of the code WAS going to be extracted to, since there was an error the system actually dumped the contents of an error into a file called bundle.tar in the folder that it would have exported to.
cat /opt/codedeploy-agent/deployment-root/7ddce865-0611-45f0-bf74-459fcf806f23/d-YK4NWBJD7/bundle.tar
This returned an error from the S3 showing that the Code Deploy was having an error downloading from S3, so I had to add access to the policy to download from S3 buckets as well
InstanceAgent::CodeDeployPlugin::CommandPoller: Missing credentials – Debug / My Fix
InstanceAgent::CodeDeployPlugin::CommandPoller: Missing credentials – Debug / My Fix
I created a Policy and Launch Configuration according to this documentation
http://docs.aws.amazon.com/codedeploy/latest/userguide/how-to-create-service-role.html
However I still received this error
InstanceAgent::CodeDeployPlugin::CommandPoller: Missing credentials - please check if this instance was started with an IAM instance profile
The problem lay somewhere in the configuration of how I setup and Launched the server, however I set about exploring the server to evaluate and confirm the ‘missing credentials’.
Using these checks I would be able to confirm that the reason behind these errors (and there fore the problem with actually deploying code):
First run this command to check to see what roles were ‘requested’
echo `wget http://169.254.169.254/latest/meta-data/iam/security-credentials/ -O - -q `
This command will return the list of Roles that were given to your server. You can then extend the request
MyRoleName
Then add the return on to the end of the URL to see the results of attempting to add the role, in my case the error was displayed
{ "Code" : "AssumeRoleUnauthorizedAccess", "Message" : "EC2 cannot assume the role MyRoleName. Please see documentation at http://docs.amazonwebservices.com/IAM/latest/UserGuide/RolesTroubleshooting.html.", "LastUpdated" : "2015-02-17T04:38:25Z" }
Basically, the EC2 was unable to assume the role MyRoleName did not have permission to Assume a Role, This could be due to the permissions of the development account that was used to start eh Scaling Group which launched the EC2. To test this,
- I login with an administrator account
- recreate the scaling group, identical
- login to the server
- run the echo `wget http://169.254.169.254/latest/meta-data/iam/security-credentials/ -O – -q ` command
This didn’t do anything, So I thought I would look into the specifics of why we have a role that should be able to be assumed, but which has a message which explains that EC2 cannot assume the role.
So I looked back at the article http://docs.aws.amazon.com/codedeploy/latest/userguide/how-to-create-service-role.html and found somewhat of an anomaly, it seems that the article suggests that we build a trust relationship with gives the ‘codedeploy.us-west-2.amazonaws.com’ service the ability to use this policy. However, that does not jive with the Messages I see in the logs defining that EC2 is not able to assume the role. So I opened the trust relationship under the Role, and Added the bolded line
{ "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Service": [ "ec2.amazonaws.com", "codedeploy.us-east-1.amazonaws.com", "codedeploy.us-west-2.amazonaws.com"
] }, "Action": "sts:AssumeRole" } ] }
Lo and Behold, it worked. Now when I run the following I get a Success Message
echo `wget http://169.254.169.254/latest/meta-data/iam/security-credentials/MyRoleName-O - -q `
So, it turns out the problem is that the article is either incorrect or was written in order to give the CodeDeploy service the ability to work on EC2, but not giving the EC2 servers access to the CodeDeploy service. ( the codedeploy.us-east-1 services are also required in order to to give the deployment group the IAM role.)
While, it was difficult to find the solution, this troubleshooting steps above are useful to help identify related or other issues, I hope you find some use from the tools
Michael Blood