sftp_hub_ecs.yaml
---

SFTP Hub CloudFormation Deployment

This CloudFormation template will build an ECS stack to support a central SFTP server with multiple EFS systems mounted.

AWSTemplateFormatVersion: '2010-09-09' Description: SFTP Hub ECS

Parameters

These are the input parameters for this template. All of these parameters must be supplied for this template to be deployed.

Parameters: UppercaseSFTPEnv: Description: Valid values are HR, SA, and ELM MinLength: '2' Type: String Default: "HR" LowercasePillar: Description: Valid values are hr, sa, and el MinLength: '2' Type: String Default: "hr" AllowedPattern: "[a-z0-9-]*" SecurityGroupCloudFormationName: Description: CloudFormation Security Group Name Type: String Default: "PeopleSoftSG" KeyName: Description: Amazon EC2 Key Pair Type: AWS::EC2::KeyPair::KeyName Default: "peoplesoft-keypair" EcsImageId:

Old way to do it

Description: The AMI Amazon built specifically for ECS Type: String

Go here to get the latest AMI, need to see if we can automate this

http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-optimized_AMI_launch_latest.html

This is 2017.03, looks like a new one comes out twice a year

Default: "ami-596d6520"

New way

Description: SSM Parameter store key for the latest ECS Optimized AMI ID Type: 'AWS::SSM::Parameter::Value<String>' Default: '/aws/service/ecs/optimized-ami/amazon-linux/recommended/image_id' AllowedValues: - '/aws/service/ecs/optimized-ami/amazon-linux/recommended/image_id' EcsInstanceType: Description: ECS Instance AWS Server Type (may need larger instance if doing backups) Type: String Default: "t3.micro" SetUpLikePrd: Description: If set to Y then will set up like a Production Environment Type: String Default: "N" AllowedValues: - Y - N Eip1: Description: Elastic IP 1 used to ensure vendors can whitelist IP Addresses Type: String Default: "none" Eip2: Description: Elastic IP 2 used to ensure vendors can whitelist IP Addresses Type: String Default: "none" LoggingLabel: Type: String Description: Used to distinguish between Prod and NonProd in exported ELB logs. AllowedValues: [prod, nonprod] Default: nonprod DockerImage: Description: 'Docker Image, eg.: eas/sftphub:2018-05-01' Default: 760232551367.dkr.ecr.us-west-2.amazonaws.com/eas/sftphub:2018-05-01 Type: String OverrideMountTarget: Description: If set to Y then will allow override of mount target path Type: String Default: "N" AllowedValues: - Y - N OverrideMountPath: Description: Set the mount target override path (eg /mosaic/HCM_NONPROD) Type: String Default: "/mosaic/HR_NONPROD" DRS3Bucket: Description: 'Disaster Recovery Bucket Name (eg edu-arizona-dr-peoplesoft-east-1)' Default: "edu-arizona-dr-peoplesoft-east-1" Type: String TagService: Description: Refers to the application (Uaccess Learning, Uaccess Employee, Uaccess Student) Type: String Default: "Uaccess Learning" TagContactNetid: Description: NetID of person most familiar with resource Type: String Default: "kellehs" TagAccountNumber: Description: Identifies the financial system account number Type: String Default: "Human Resources Systems" TagSubAccount: Description: Identifies the financial system subaccount number Type: String Default: "Uaccess Learning" TagTicketNumber: Description: Jira Ticket Number Type: String Default: "CLOUD-15" #Next is the Conditions section, these will be used to build additional #infrastructure for production Conditions: ThisIsProd: !Equals [!Ref SetUpLikePrd, "Y"] ThisIsELM: !Equals [!Ref UppercaseSFTPEnv, "ELM"] NoEips: !Equals [!Ref Eip1, "none"] YesEips: !Not [!Equals [!Ref Eip1, "none"]] OverridePath: !Equals [!Ref OverrideMountTarget, "Y"] #Mappings Section Mappings: EFSID: #nonprod account number "415418166582": "el": "fs-e15fa548" "hr": "fs-e62ef24f" "sa": "fs-a1a5c208" #prod account number "242275737326": "el": "fs-6e6facc7" "hr": "fs-e64d914f" "sa": "fs-a3117b0a"

Metadata

Metadata is mostly for organizing and presenting Parameters in a better way when using CloudFormation in the AWS Web UI.

Metadata: AWS::CloudFormation::Interface: ParameterGroups: - Label: default: Application Information Parameters: - UppercaseSFTPEnv - LowercasePillar - Label: default: Instance Settings Parameters: - SecurityGroupCloudFormationName - KeyName - EcsImageId - EcsInstanceType - SetUpLikePrd - Label: default: Load Balancer Settings Parameters: - Eip1 - Eip2 - LoggingLabel - Label: default: Application Settings Parameters: - DockerImage - OverrideMountTarget - OverrideMountPath - DRS3Bucket - Label: default: Tags Parameters: - TagService - TagContactNetid - TagAccountNumber - TagSubAccount - TagTicketNumber

Resources

These are all of the actual AWS resources created for this application.

Resources:

Instance Role

This is the IAM role that will be applied to the ECS Instances. Any AWS specific permissions that the node might need should be defined here.

EnvInstanceRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: - ec2.amazonaws.com Action: - sts:AssumeRole Path: "/" ManagedPolicyArns: - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role Policies: - PolicyName: "dr-backup-access" PolicyDocument: Version: '2012-10-17' Statement: - Sid: Stmt1452033379000 Effect: Allow Action: - s3:ListBucket - s3:ListBucketVersions Resource: - !Sub "arn:aws:s3:::${DRS3Bucket}" - Sid: Stmt1452033379200 Effect: Allow Action: - s3:PutObject - s3:GetObject - s3:DeleteObject - s3:PutObjectAcl Resource: - !Sub "arn:aws:s3:::${DRS3Bucket}/*" - PolicyName: "sns-notifications" PolicyDocument: Version: '2012-10-17' Statement: - Sid: Stmt1452033379100 Effect: Allow Action: - sns:publish Resource: - !ImportValue fdn-logging-alarm-topic

Instance Profile

This is just a little construct to connect a set of roles together into a profile. The profile is referenced by ec2 instances.

EnvInstanceProfile: Type: AWS::IAM::InstanceProfile Properties: Path: "/" Roles: - !Ref EnvInstanceRole

Instance Security Group

Security group for the host nodes themselves. Needs to permit incoming traffice from the ELB, and any other authorized incoming sources.

InstanceSecurityGroup: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: Allow Load Balancer and SSH to host VpcId: Fn::ImportValue: !Sub "${SecurityGroupCloudFormationName}-VPCID" SecurityGroupIngress:

Opening up to all traffic as Network LBs don't allow SG on external side. This is normal with NLB, as security is enforced at the host.

- IpProtocol: tcp FromPort: '31000' ToPort: '61000' CidrIp: 0.0.0.0/0 Description: Dynamic port NLB - IpProtocol: tcp FromPort: '22' ToPort: '22' CidrIp: 10.138.2.0/24 Description: Mosaic VPN-1 - IpProtocol: tcp FromPort: '22' ToPort: '22' CidrIp: 150.135.241.0/24 Description: Mosaic VPN-2 - IpProtocol: tcp FromPort: '22' ToPort: '22' CidrIp: 150.135.112.0/24 Description: InfraDev VPN - IpProtocol: tcp FromPort: '22' ToPort: '22' CidrIp: 128.196.130.211/32 Description: ben.uits bastion Tags: - Key: service Value: !Ref TagService - Key: Name Value: !Sub "${LowercasePillar}-sftp-hub-inst-sg" - Key: environment Value: !Sub "${LowercasePillar}-sftp-hub" - Key: contactnetid Value: !Ref TagContactNetid - Key: accountnumber Value: !Ref TagAccountNumber - Key: ticketnumber Value: !Ref TagTicketNumber - Key: subaccount Value: !Ref TagSubAccount

Elastic Load Balancer (ELB)

Create this LB if we are using Elastic IPs so we can provide vendors with a set of static IPs

SftpNetworkLoadBalancerEip: Type: AWS::ElasticLoadBalancingV2::LoadBalancer Condition: YesEips Properties: Name: !Sub "peoplesoft-${LowercasePillar}-sftp-network-lb" Type: "network" Scheme: internet-facing ##Network Load Balancers don't support logging to S3 buckets #LoadBalancerAttributes: #- Key: access_logs.s3.enabled

Value: true

#- Key: access_logs.s3.bucket

Value: edu.arizona.iso.elb.logs

#- Key: access_logs.s3.prefix

Value: !Sub "ps${LowercasePillar}-${LoggingLabel}-sftp"

#Subnets:
  • Fn::ImportValue: !Sub "${SecurityGroupCloudFormationName}-PubSubNet1"
  • Fn::ImportValue: !Sub "${SecurityGroupCloudFormationName}-PubSubNet2"
SubnetMappings: - AllocationId: !Ref Eip1 SubnetId: Fn::ImportValue: !Sub "${SecurityGroupCloudFormationName}-PubSubNet1" - AllocationId: !Ref Eip2 SubnetId: Fn::ImportValue: !Sub "${SecurityGroupCloudFormationName}-PubSubNet2" Tags: - Key: service Value: !Ref TagService - Key: Name Value: !Sub "${LowercasePillar}-sftp-hub" - Key: environment Value: !Sub "${LowercasePillar}-sftp-hub" - Key: contactnetid Value: !Ref TagContactNetid - Key: accountnumber Value: !Ref TagAccountNumber - Key: ticketnumber Value: !Ref TagTicketNumber - Key: subaccount Value: !Ref TagSubAccount

Create this LB if we are NOT using Elastic IPs

SftpNetworkLoadBalancer: Type: AWS::ElasticLoadBalancingV2::LoadBalancer Condition: NoEips Properties: Name: !Sub "peoplesoft-${LowercasePillar}-sftp-network-lb" Type: "network" Scheme: internet-facing ##Network Load Balancers don't support logging to S3 buckets #LoadBalancerAttributes: #- Key: access_logs.s3.enabled

Value: true

#- Key: access_logs.s3.bucket

Value: edu.arizona.iso.elb.logs

#- Key: access_logs.s3.prefix

Value: !Sub "ps${LowercasePillar}-${LoggingLabel}-sftp"

Subnets: - Fn::ImportValue: !Sub "${SecurityGroupCloudFormationName}-PubSubNet1" - Fn::ImportValue: !Sub "${SecurityGroupCloudFormationName}-PubSubNet2" Tags: - Key: service Value: !Ref TagService - Key: Name Value: !Sub "${LowercasePillar}-sftp-hub" - Key: environment Value: !Sub "${LowercasePillar}-sftp-hub" - Key: contactnetid Value: !Ref TagContactNetid - Key: accountnumber Value: !Ref TagAccountNumber - Key: ticketnumber Value: !Ref TagTicketNumber - Key: subaccount Value: !Ref TagSubAccount #ELB Target group for SFTP ECS Cluster SftpELBV2Tg: Type: "AWS::ElasticLoadBalancingV2::TargetGroup" Properties: Name: !Sub "peoplesoft-${LowercasePillar}-sftp-tg" HealthCheckIntervalSeconds: "30" HealthCheckProtocol: "TCP" HealthCheckPort: "traffic-port" HealthCheckTimeoutSeconds: "10" HealthyThresholdCount: "2" UnhealthyThresholdCount: "2" Port: 2022 Protocol: "TCP" TargetGroupAttributes: - Key: "deregistration_delay.timeout_seconds" Value: "300" VpcId: Fn::ImportValue: !Sub "${SecurityGroupCloudFormationName}-VPCID" #ELB Listeners for SFTP Network LB SftpELBListener: Type: "AWS::ElasticLoadBalancingV2::Listener" Properties: DefaultActions: - Type: "forward" TargetGroupArn: !Ref "SftpELBV2Tg" LoadBalancerArn: !If [YesEips, !Ref SftpNetworkLoadBalancerEip, !Ref SftpNetworkLoadBalancer] Port: "22" Protocol: "TCP"

Route53 DNS Record

Create a DNS entry in Route53 for this environment. This creates a CNAME pointing at the DNS name of the Load Balancer.

AppDnsRecord: Type: AWS::Route53::RecordSet Properties:

Append a period after the hosted zone DNS name

HostedZoneName: "Fn::Sub": - "${HostedZoneName}." - { HostedZoneName: !ImportValue peoplesoft-route53-dns } Name: "Fn::Sub": - "sftp.${LowercasePillar}.${HostedZoneName}." - { HostedZoneName: !ImportValue peoplesoft-route53-dns } Type: CNAME TTL: '900' ResourceRecords: - !If [YesEips, !GetAtt SftpNetworkLoadBalancerEip.DNSName, !GetAtt SftpNetworkLoadBalancer.DNSName] #If this is ELM we need to create a legacy Route53 entry #sftp.peoplesoft-sftp-hub.(hosted zone) (i.e. sftp.peoplesoft-sftp-hub.ps-prod-aws.arizona.edu) LegacyDnsRecord: Type: AWS::Route53::RecordSet Condition: ThisIsELM Properties:

Append a period after the hosted zone DNS name

HostedZoneName: "Fn::Sub": - "${HostedZoneName}." - { HostedZoneName: !ImportValue peoplesoft-route53-dns } Name: "Fn::Sub": - "sftp.peoplesoft-sftp-hub.${HostedZoneName}." - { HostedZoneName: !ImportValue peoplesoft-route53-dns } Type: CNAME TTL: '900' ResourceRecords: - !If [YesEips, !GetAtt SftpNetworkLoadBalancerEip.DNSName, !GetAtt SftpNetworkLoadBalancer.DNSName] #Need to create a LogGroup in order for the ECS service to log details of the build #If this does not exist the ECS Service will not come up EcsLogGroup: Type: AWS::Logs::LogGroup Properties: LogGroupName: !Sub "${UppercaseSFTPEnv}-SFTP-Hub-LG" RetentionInDays: 30 #Launch Config for the Auto Scaling Group for the ECS Cluster EcsInstanceLc: Type: AWS::AutoScaling::LaunchConfiguration Properties: ImageId: !Ref EcsImageId InstanceType: !Ref EcsInstanceType AssociatePublicIpAddress: false IamInstanceProfile: !Ref EnvInstanceProfile KeyName: !Ref KeyName SecurityGroups: - !Ref InstanceSecurityGroup - Fn::ImportValue: !Sub "${SecurityGroupCloudFormationName}-ELMEFSSG" - Fn::ImportValue: !Sub "${SecurityGroupCloudFormationName}-HREFSSG" - Fn::ImportValue: !Sub "${SecurityGroupCloudFormationName}-SAEncryptedEFSSG" - Fn::ImportValue: !Sub "${SecurityGroupCloudFormationName}-SshSg" BlockDeviceMappings: - DeviceName: "/dev/xvdcz" Ebs: VolumeSize: "22" VolumeType: "gp2" UserData: Fn::Base64: !Sub | #!/bin/bash echo ECS_CLUSTER=${UppercaseSFTPEnv}SFTP >> /etc/ecs/ecs.config #Auto Scaling Group for Web/App/Batch #Will be used in all non-prod environments, if this is prod this will not be created EcsInstanceAsg: Type: AWS::AutoScaling::AutoScalingGroup DependsOn: EcsCluster Properties: VPCZoneIdentifier: - Fn::ImportValue: !Sub "${SecurityGroupCloudFormationName}-PrivSubNet1" - Fn::ImportValue: !Sub "${SecurityGroupCloudFormationName}-PrivSubNet2" LaunchConfigurationName: !Ref EcsInstanceLc MinSize: '0' #If this is production we'll need to instances for HA, for nonprod only 1 MaxSize: !If [ThisIsProd,"2","1"] DesiredCapacity: !If [ThisIsProd,"2","1"] TargetGroupARNs: - !Ref SftpELBV2Tg Tags: - Key: Name Value: !Sub "ECS Instance - ${AWS::StackName}" PropagateAtLaunch: 'true' - Key: Description Value: "This instance is the part of the Auto Scaling group which was created through CloudFormation" PropagateAtLaunch: 'true' - Key: service Value: !Ref TagService PropagateAtLaunch: 'true' - Key: environment Value: !Sub "${LowercasePillar}-sftp-hub" PropagateAtLaunch: 'true' - Key: contactnetid Value: !Ref TagContactNetid PropagateAtLaunch: 'true' - Key: accountnumber Value: !Ref TagAccountNumber PropagateAtLaunch: 'true' - Key: ticketnumber Value: !Ref "TagTicketNumber" PropagateAtLaunch: 'true' - Key: subaccount Value: !Ref TagSubAccount PropagateAtLaunch: 'true' #ECS Cluster EcsCluster: Type: "AWS::ECS::Cluster" #Need to make sure the LB is created before the ECS cluster is created Properties: ClusterName: !Sub "${UppercaseSFTPEnv}SFTP" #ECS Task Definition EcsTask: Type: "AWS::ECS::TaskDefinition" Properties: Family: !Sub "${UppercaseSFTPEnv}-SFTP" NetworkMode: "bridge" ContainerDefinitions: - Name: !Sub "${UppercaseSFTPEnv}-SFTP" Essential: "true" Image: !Ref DockerImage PortMappings: - HostPort: "0" ContainerPort: "22" Protocol: "tcp" Hostname: !Sub "${LowercasePillar}-sftp" Cpu: "200" MemoryReservation: "512" Privileged: "true" Environment: - Name: "SFTP_ENV" Value: !Sub "${UppercaseSFTPEnv}" - Name: "SFTP_MOUNTS" Value: Fn::Sub: - "${ThisEFSID}.efs.${AWS::Region}.amazonaws.com:/=${MountPath}" - ThisEFSID: Fn::FindInMap: [EFSID,!Ref "AWS::AccountId",!Ref "LowercasePillar"] MountPath: Fn::If: [OverridePath, !Ref OverrideMountPath, !Sub "/mosaic/${UppercaseSFTPEnv}"] - Name: "SNS_NOTIFICATION_TOPIC" Value: !ImportValue fdn-logging-alarm-topic - Name: "BackupSourcePath" Value: Fn::Sub: - "${MountPath}" - MountPath: Fn::If: [OverridePath, !Ref OverrideMountPath, !Sub "/mosaic/${UppercaseSFTPEnv}"] - Name: "S3DRBackupURI" Value: !Sub "s3://${DRS3Bucket}/${LowercasePillar}${LoggingLabel}" LogConfiguration: LogDriver: "awslogs" Options: awslogs-group: !Ref "EcsLogGroup" awslogs-region: !Ref "AWS::Region" awslogs-stream-prefix: "SFTP-HUB"

ECS Service

EcsService: Type: "AWS::ECS::Service" #Waiting for the DNS record to be created beause we know the LB has been #created if the DNS record is created, and the LB needs to be created #before the ECS service is created DependsOn: - EcsInstanceAsg - AppDnsRecord Properties: ServiceName: !Sub "${UppercaseSFTPEnv}-SFTP" Cluster: !Ref EcsCluster TaskDefinition: !Ref EcsTask DesiredCount: !If [ThisIsProd,"2","1"] Role: !Ref EcsServiceRole LoadBalancers: - ContainerName: !Sub "${UppercaseSFTPEnv}-SFTP" ContainerPort: "22" TargetGroupArn: !Ref SftpELBV2Tg #For now we will spread across AZs PlacementStrategies: - Field: "attribute:ecs.availability-zone" Type: "spread" #Will take the defaults for this right now, may not apply to PeopleSoft DeploymentConfiguration: MaximumPercent: "200" #PlacementConstraints:
  • PlacementConstraints, ...

ECS Service Role

EcsServiceRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: - ecs.amazonaws.com Action: - sts:AssumeRole Path: "/" ManagedPolicyArns: - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole

Outputs

Output values that can be viewed from the AWS CloudFormation console.

Outputs: LoadBalancerDNS: Value: !If [YesEips, !GetAtt SftpNetworkLoadBalancerEip.DNSName, !GetAtt SftpNetworkLoadBalancer.DNSName] Export: Name: !Sub "${AWS::StackName}-lb-dns" SftpEndpointDNS: Value: !Ref AppDnsRecord Export: Name: !Sub "${AWS::StackName}-dns"