sftp_hub_ecs.yaml
---

SFTP Hub CloudFormation Deployment

This CloudFormation template will build an ECS stack to support a central SFTP server with multiple EFS systems mounted.

AWSTemplateFormatVersion: '2010-09-09' Description: SFTP Hub ECS

Parameters

These are the input parameters for this template. All of these parameters must be supplied for this template to be deployed.

Parameters: AppSlug: MinLength: '3' Type: String Description: Short application slug, ie 'kfs'. Lowercase letters, numbers and dashes only AllowedPattern: "[a-z0-9-]*" EnvSlug: MinLength: '2' Type: String Description: Short environment slug, ie 'dev', or 'markdev'. Lowercase letters, numbers and dashes only AllowedPattern: "[a-z0-9]*" VPC: Type: AWS::EC2::VPC::Id Description: The VPC in which to deploy this stack KeyName: Description: Amazon EC2 Key Pair Type: AWS::EC2::KeyPair::KeyName Default: "kuali-prod-keypair" EcsImageId:

Old way to do it

Description: The AMI Amazon built specifically for ECS Type: String

Go here to get the latest AMI, need to see if we can automate this

http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-optimized_AMI_launch_latest.html

This is 2017.03, looks like a new one comes out twice a year

Default: "ami-596d6520"

New way

Description: SSM Parameter store key for the latest ECS Optimized AMI ID Type: 'AWS::SSM::Parameter::Value<String>' Default: '/aws/service/ecs/optimized-ami/amazon-linux/recommended/image_id' AllowedValues: - '/aws/service/ecs/optimized-ami/amazon-linux/recommended/image_id' InstanceSubnets: Type: List<AWS::EC2::Subnet::Id> Description: The private subnets for the application (2 or more) EcsInstanceType: Description: ECS Instance AWS Server Type (may need larger instance if doing backups) Type: String Default: "t3.micro" SetUpLikePrd: Description: If set to Y then will set up like a Production Environment Type: String Default: "N" AllowedValues: - Y - N Eip1: Description: Elastic IP 1 used to ensure vendors can whitelist IP Addresses Type: String Default: "none" Eip2: Description: Elastic IP 2 used to ensure vendors can whitelist IP Addresses Type: String Default: "none" LBSubnets: Type: List<AWS::EC2::Subnet::Id> Description: private subnets for load balancer (2 or more) PrivateLB: Description: "Private subnet Load Balancer (Y/N - default: N)" Type: String AllowedValues: [N, Y] Default: N Route53StackName: Description: Name of the Route53 CloudFormation stack for this account Type: String Default: "kfs-route53" Route53ZoneName: Description: (Optional) Name of the DNS zone for this deployment, instead of and overrides Route53StackName Type: String Default: "none" LoggingLabel: Type: String Description: Used to distinguish between Prod and NonProd in exported ELB logs. AllowedValues: [prod, nonprod] Default: nonprod DockerImage: Description: 'Docker Image, eg.: eas/sftphub:2018-05-01' Default: 740525297805.dkr.ecr.us-west-2.amazonaws.com/eas/sftphub:2018-05-01 Type: String EFSStackName: MinLength: '2' Type: String Description: Name of the EFS CloudFormation Stack Default: kfs-efs SFTPBranchName: Description: 'Branch of the SFTP logic to use, eg. KFS_PROD' Default: KFS_PROD Type: String OverrideMountTarget: Description: If set to Y then will allow override of mount target path Type: String Default: "N" AllowedValues: - Y - N OverrideMountPath: Description: Set the mount target override path (eg /mosaic/KFS7_NONPROD) Type: String Default: "/mosaic/KFS7_NONPROD" DRS3Bucket: Description: 'Disaster Recovery Bucket Name' Default: "edu-arizona-dr-kuali" Type: String TagService: Description: Refers to the application (Uaccess Learning, Uaccess Employee, Uaccess Student) Type: String Default: "Uaccess Financials" TagEnvironment: Description: Type of environment that is using this resource, such as 'dev', 'tst', 'prd'. Type: String TagContactNetid: Description: NetID of person most familiar with resource Type: String Default: "fischerm" TagAccountNumber: Description: Identifies the financial system account number Type: String Default: "1192620" TagSubAccount: Description: Identifies the financial system subaccount number Type: String Default: "12AWS" TagTicketNumber: Description: Jira Ticket Number Type: String Default: "UAF-6443" TagApplication: Description: Provided Application Type: String Default: "sftp"

Metadata

Metadata is mostly for organizing and presenting Parameters in a better way when using CloudFormation in the AWS Web UI.

Metadata: AWS::CloudFormation::Interface: ParameterGroups: - Label: default: Application Information Parameters: - AppSlug - EnvSlug - VPC - Label: default: Instance Settings Parameters: - KeyName - EcsImageId - InstanceSubnets - EcsInstanceType - SetUpLikePrd - Label: default: Load Balancer Settings Parameters: - Eip1 - Eip2 - LBSubnets - PrivateLB - Route53StackName - Route53ZoneName - LoggingLabel - Label: default: Application Settings Parameters: - DockerImage - EFSStackName - SFTPBranchName - OverrideMountTarget - OverrideMountPath - DRS3Bucket - Label: default: Tags Parameters: - TagService - TagName - TagEnvironment - TagContactNetid - TagAccountNumber - TagSubAccount - TagTicketNumber - TagApplication #Next is the Conditions section, these will be used to build additional #infrastructure for production Conditions: ThisIsProd: !Equals [!Ref SetUpLikePrd, "Y"] NoEips: !Equals [!Ref Eip1, "none"] YesEips: !Not [!Equals [!Ref Eip1, "none"]] IsPrivateLB: !Equals [!Ref PrivateLB, "Y"] OverridePath: !Equals [!Ref OverrideMountTarget, "Y"] UseRoute53CF: !Equals [!Ref Route53ZoneName, "none"]

Resources

These are all of the actual AWS resources created for this application.

Resources:

Instance Role

This is the IAM role that will be applied to the ECS Instances. Any AWS specific permissions that the node might need should be defined here.

EnvInstanceRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: - ec2.amazonaws.com Action: - sts:AssumeRole Path: "/" ManagedPolicyArns: - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role Policies: - PolicyName: "dr-backup-access" PolicyDocument: Version: '2012-10-17' Statement: - Sid: Stmt1452033379000 Effect: Allow Action: - s3:ListBucket - s3:ListBucketVersions - s3:PutObject - s3:GetObject - s3:DeleteObject - s3:PutObjectAcl Resource: - !Sub "arn:aws:s3:::${DRS3Bucket}" - !Sub "arn:aws:s3:::${DRS3Bucket}/*" - PolicyName: "sns-notifications" PolicyDocument: Version: '2012-10-17' Statement: - Sid: Stmt1452033379100 Effect: Allow Action: - sns:publish Resource: - !ImportValue fdn-logging-alarm-topic

Instance Profile

This is just a little construct to connect a set of roles together into a profile. The profile is referenced by ec2 instances.

EnvInstanceProfile: Type: AWS::IAM::InstanceProfile Properties: Path: "/" Roles: - !Ref EnvInstanceRole

Instance Security Group

Security group for the host nodes themselves. Needs to permit incoming traffice from the ELB, and any other authorized incoming sources.

InstanceSecurityGroup: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: Allow Load Balancer and SSH to host VpcId: !Ref VPC SecurityGroupIngress:

Opening up to all traffic as Network LBs don't allow SG on external side. This is normal with NLB, as security is enforced at the host.

- IpProtocol: tcp FromPort: '31000' ToPort: '61000' CidrIp: 0.0.0.0/0 Description: Dynamic port NLB - IpProtocol: tcp FromPort: '22' ToPort: '22' CidrIp: 10.138.2.0/24 Description: Mosaic VPN-1 - IpProtocol: tcp FromPort: '22' ToPort: '22' CidrIp: 150.135.241.0/24 Description: Mosaic VPN-2 - IpProtocol: tcp FromPort: '22' ToPort: '22' CidrIp: 150.135.112.0/24 Description: InfraDev VPN - IpProtocol: tcp FromPort: '22' ToPort: '22' CidrIp: 128.196.130.211/32 Description: ben.uits bastion Tags: - Key: service Value: !Ref TagService - Key: Name Value: !Sub "${AppSlug}-${EnvSlug}-sftp-hub-inst-sg" - Key: environment Value: !Ref TagEnvironment - Key: contactnetid Value: !Ref TagContactNetid - Key: accountnumber Value: !Ref TagAccountNumber - Key: ticketnumber Value: !Ref TagTicketNumber - Key: subaccount Value: !Ref TagSubAccount - Key: application Value: !Ref TagApplication

Elastic Load Balancer (ELB)

Create this LB if we are using Elastic IPs so we can provide vendors with a set of static IPs

SftpNetworkLoadBalancerEip: Type: AWS::ElasticLoadBalancingV2::LoadBalancer Condition: YesEips Properties: Name: !Sub "${AppSlug}-${EnvSlug}-sftp-network-lb" Type: "network" Scheme: internet-facing SubnetMappings: - AllocationId: !Ref Eip1 SubnetId: !ImportValue kuali-vpc-public-subnet-a - AllocationId: !Ref Eip2 SubnetId: !ImportValue kuali-vpc-public-subnet-b Tags: - Key: service Value: !Ref TagService - Key: Name Value: !Sub "${AppSlug}-${EnvSlug}-sftp-hub" - Key: environment Value: !Ref TagEnvironment - Key: contactnetid Value: !Ref TagContactNetid - Key: accountnumber Value: !Ref TagAccountNumber - Key: ticketnumber Value: !Ref TagTicketNumber - Key: subaccount Value: !Ref TagSubAccount - Key: application Value: !Ref TagApplication

Create this LB if we are NOT using Elastic IPs

SftpNetworkLoadBalancer: Type: AWS::ElasticLoadBalancingV2::LoadBalancer Condition: NoEips Properties: Name: !Sub "${AppSlug}-${EnvSlug}-sftp-nlb" Type: "network" Scheme: !If - IsPrivateLB - internal - internet-facing Subnets: !Ref LBSubnets Tags: - Key: service Value: !Ref TagService - Key: Name Value: !Sub "${AppSlug}-${EnvSlug}-sftp-nlb" - Key: environment Value: !Ref TagEnvironment - Key: contactnetid Value: !Ref TagContactNetid - Key: accountnumber Value: !Ref TagAccountNumber - Key: ticketnumber Value: !Ref TagTicketNumber - Key: subaccount Value: !Ref TagSubAccount - Key: application Value: !Ref TagApplication #ELB Target group for SFTP ECS Cluster SftpELBV2Tg: Type: "AWS::ElasticLoadBalancingV2::TargetGroup" Properties: Name: !Sub "${AppSlug}-${EnvSlug}-sftp-tg" HealthCheckIntervalSeconds: "30" HealthCheckProtocol: "TCP" HealthCheckPort: "traffic-port" HealthCheckTimeoutSeconds: "10" HealthyThresholdCount: "2" UnhealthyThresholdCount: "2" Port: 2022 Protocol: "TCP" TargetGroupAttributes: - Key: "deregistration_delay.timeout_seconds" Value: "300" VpcId: !Ref VPC #ELB Listeners for SFTP Network LB SftpELBListener: Type: "AWS::ElasticLoadBalancingV2::Listener" Properties: DefaultActions: - Type: "forward" TargetGroupArn: !Ref "SftpELBV2Tg" LoadBalancerArn: !If [YesEips, !Ref SftpNetworkLoadBalancerEip, !Ref SftpNetworkLoadBalancer] Port: "22" Protocol: "TCP"

Route53 DNS Record

Create a DNS entry in Route53 for this environment. This creates a CNAME pointing at the DNS name of the Load Balancer.

AppDnsRecord: Type: AWS::Route53::RecordSet Properties:

Append a period after the hosted zone DNS name

HostedZoneName: "Fn::Sub": - "${HostedZoneName}." - HostedZoneName: !If - UseRoute53CF - Fn::ImportValue: !Sub "${Route53StackName}-dns" - !Ref Route53ZoneName Name: "Fn::Sub": - "sftp.${AppSlug}${EnvSlug}.${HostedZoneName}." - HostedZoneName: !If - UseRoute53CF - Fn::ImportValue: !Sub "${Route53StackName}-dns" - !Ref Route53ZoneName Type: CNAME TTL: '900' ResourceRecords: - !If [YesEips, !GetAtt SftpNetworkLoadBalancerEip.DNSName, !GetAtt SftpNetworkLoadBalancer.DNSName] #Need to create a LogGroup in order for the ECS service to log details of the build #If this does not exist the ECS Service will not come up EcsLogGroup: Type: AWS::Logs::LogGroup Properties: LogGroupName: !Sub "${AppSlug}-${EnvSlug}-sftp-lg" RetentionInDays: 30 #Launch Config for the Auto Scaling Group for the ECS Cluster EcsInstanceLc: Type: AWS::AutoScaling::LaunchConfiguration Properties: ImageId: !Ref EcsImageId InstanceType: !Ref EcsInstanceType AssociatePublicIpAddress: false IamInstanceProfile: !Ref EnvInstanceProfile KeyName: !Ref KeyName SecurityGroups: - !Ref InstanceSecurityGroup - Fn::ImportValue: !Sub "${EFSStackName}-target-sg" BlockDeviceMappings: - DeviceName: "/dev/xvdcz" Ebs: VolumeSize: "22" VolumeType: "gp2" UserData: Fn::Base64: !Sub | #!/bin/bash echo ECS_CLUSTER=${AppSlug}-${EnvSlug}-SFTP >> /etc/ecs/ecs.config #Auto Scaling Group for Web/App/Batch #Will be used in all non-prod environments, if this is prod this will not be created EcsInstanceAsg: Type: AWS::AutoScaling::AutoScalingGroup DependsOn: EcsCluster Properties: VPCZoneIdentifier: !Ref InstanceSubnets LaunchConfigurationName: !Ref EcsInstanceLc MinSize: '0' #If this is production we'll need to instances for HA, for nonprod only 1 MaxSize: !If [ThisIsProd,"2","1"] DesiredCapacity: !If [ThisIsProd,"2","1"] TargetGroupARNs: - !Ref SftpELBV2Tg Tags: - Key: Name Value: !Sub "${AWS::StackName} ECS Host" PropagateAtLaunch: 'true' - Key: Description Value: "This instance is the part of the Auto Scaling group which was created through CloudFormation" PropagateAtLaunch: 'true' - Key: service Value: !Ref TagService PropagateAtLaunch: 'true' - Key: environment Value: !Ref TagEnvironment PropagateAtLaunch: 'true' - Key: contactnetid Value: !Ref TagContactNetid PropagateAtLaunch: 'true' - Key: accountnumber Value: !Ref TagAccountNumber PropagateAtLaunch: 'true' - Key: ticketnumber Value: !Ref TagTicketNumber PropagateAtLaunch: 'true' - Key: subaccount Value: !Ref TagSubAccount PropagateAtLaunch: 'true' - Key: application Value: !Ref TagApplication PropagateAtLaunch: 'true' #ECS Cluster EcsCluster: Type: "AWS::ECS::Cluster" #Need to make sure the LB is created before the ECS cluster is created Properties: ClusterName: !Sub "${AppSlug}-${EnvSlug}-SFTP" #ECS Task Definition EcsTask: Type: "AWS::ECS::TaskDefinition" Properties: Family: !Sub "${AppSlug}-${EnvSlug}-SFTP" NetworkMode: "bridge" ContainerDefinitions: - Name: !Sub "${AppSlug}-${EnvSlug}-SFTP" Essential: "true" Image: !Ref DockerImage PortMappings: - HostPort: "0" ContainerPort: "22" Protocol: "tcp" Hostname: !Sub "${AppSlug}-${EnvSlug}-sftp" Cpu: "200" MemoryReservation: "512" Privileged: "true" Environment: - Name: "SFTP_ENV" Value: !Ref SFTPBranchName - Name: "SFTP_MOUNTS" Value: Fn::Sub: - "${EFSID}.efs.${AWS::Region}.amazonaws.com:/=${MountPath}" - EFSID: Fn::ImportValue: !Sub "${EFSStackName}-fs-id" MountPath: Fn::If: [OverridePath, !Ref OverrideMountPath, !Sub "/mosaic/${SFTPBranchName}"] - Name: "SNS_NOTIFICATION_TOPIC" Value: !ImportValue fdn-logging-alarm-topic - Name: "BackupSourcePath" Value: Fn::Sub: - "${MountPath}" - MountPath: Fn::If: [OverridePath, !Ref OverrideMountPath, !Sub "/mosaic/${SFTPBranchName}"] - Name: "S3DRBackupURI" Value: !Sub "s3://${DRS3Bucket}/${AppSlug}${EnvSlug}" LogConfiguration: LogDriver: "awslogs" Options: awslogs-group: !Ref "EcsLogGroup" awslogs-region: !Ref "AWS::Region" awslogs-stream-prefix: "SFTP-HUB"

ECS Service

EcsService: Type: "AWS::ECS::Service" #Waiting for the DNS record to be created beause we know the LB has been #created if the DNS record is created, and the LB needs to be created #before the ECS service is created DependsOn: - EcsInstanceAsg - AppDnsRecord Properties: ServiceName: !Sub "${AppSlug}-${EnvSlug}-SFTP" Cluster: !Ref EcsCluster TaskDefinition: !Ref EcsTask DesiredCount: !If [ThisIsProd,"2","1"] Role: !Ref EcsServiceRole LoadBalancers: - ContainerName: !Sub "${AppSlug}-${EnvSlug}-SFTP" ContainerPort: "22" TargetGroupArn: !Ref SftpELBV2Tg #For now we will spread across AZs PlacementStrategies: - Field: "attribute:ecs.availability-zone" Type: "spread" DeploymentConfiguration: MaximumPercent: "200" #PlacementConstraints:
  • PlacementConstraints, ...

ECS Service Role

EcsServiceRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: - ecs.amazonaws.com Action: - sts:AssumeRole Path: "/" ManagedPolicyArns: - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole

Outputs

Output values that can be viewed from the AWS CloudFormation console.

Outputs: LoadBalancerDNS: Value: !If [YesEips, !GetAtt SftpNetworkLoadBalancerEip.DNSName, !GetAtt SftpNetworkLoadBalancer.DNSName] Export: Name: !Sub "${AWS::StackName}-lb-dns" SftpEndpointDNS: Value: !Ref AppDnsRecord Export: Name: !Sub "${AWS::StackName}-dns"