sftp_hub_ecs.yaml
---

SFTP Hub CloudFormation Deployment

This CloudFormation template will build an ECS stack to support a central SFTP server with multiple EFS systems mounted.

AWSTemplateFormatVersion: '2010-09-09' Description: SFTP Hub ECS

Parameters

These are the input parameters for this template. All of these parameters must be supplied for this template to be deployed.

Parameters: UppercaseSFTPEnv: Description: Valid values are HR, SA, and ELM MinLength: '2' Type: String Default: "HR" LowercasePillar: Description: Valid values are hr, sa, and el MinLength: '2' Type: String Default: "hr" AllowedPattern: "[a-z0-9-]*" SecurityGroupCloudFormationName: Description: CloudFormation Security Group Name Type: String Default: "PeopleSoftSG" KeyName: Description: Amazon EC2 Key Pair Type: AWS::EC2::KeyPair::KeyName Default: "peoplesoft-keypair" EcsImageId:

Old way to do it

Description: The AMI Amazon built specifically for ECS Type: String

Go here to get the latest AMI, need to see if we can automate this

http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-optimized_AMI_launch_latest.html

This is 2017.03, looks like a new one comes out twice a year

Default: "ami-596d6520"

New way

Description: SSM Parameter store key for the latest ECS Optimized AMI ID Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id> Default: '/aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id' AllowedValues: - '/aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id' EcsInstanceType: Description: ECS Instance AWS Server Type (may need larger instance if doing backups) Type: String Default: "t3.micro" SetUpLikePrd: Description: If set to Y then will set up like a Production Environment Type: String Default: "N" AllowedValues: - Y - N Eip1: Description: Elastic IP 1 used to ensure vendors can whitelist IP Addresses Type: String Default: "none" Eip2: Description: Elastic IP 2 used to ensure vendors can whitelist IP Addresses Type: String Default: "none" LoggingLabel: Type: String Description: Used to distinguish between Prod and NonProd in exported ELB logs. AllowedValues: [prod, nonprod] Default: nonprod DockerImage: Description: 'Docker Image, eg.: eas/sftphub:2018-05-01' Default: 722748364533.dkr.ecr.us-west-2.amazonaws.com/eas/sftphub:2020-02-12 Type: String OverrideMountTarget: Description: If set to Y then will allow override of mount target path Type: String Default: "N" AllowedValues: - Y - N OverrideMountPath: Description: Set the mount target override path (eg /mosaic/HCM_NONPROD) Type: String Default: "/mosaic/HR_NONPROD" OverrideEFS: Description: If set to Y then will allow override of EFS mount ID (eg for DR deployment) Type: String Default: "N" AllowedValues: - Y - N OverrideEFSVal: Description: Set the EFS mount ID override (eg fs-abcd1234) Type: String DRS3Bucket: Description: Disaster Recovery Bucket Name (eg edu-arizona-dr-peoplesoft-east-1) Default: "edu-arizona-dr-peoplesoft-east-1" Type: String SNSNotificationImport: Description: Output field to import as the SNS topic for notifications. Default none. Type: String Default: "" SNSNotificationEmail: Description: Specify email address for new SNS topic instead of importing an external SNS topic. Default none. Type: String Default: "" TagService: Description: Refers to the application (Uaccess Learning, Uaccess Employee, Uaccess Student) Type: String Default: "Uaccess Learning" TagContactNetid: Description: NetID of person most familiar with resource Type: String Default: "kellehs" TagAccountNumber: Description: Identifies the financial system account number Type: String Default: "Human Resources Systems" TagSubAccount: Description: Identifies the financial system subaccount number Type: String Default: "Uaccess Learning" TagTicketNumber: Description: Jira Ticket Number Type: String Default: "CLOUD-15" #Next is the Conditions section, these will be used to build additional #infrastructure for production Conditions: ThisIsProd: !Equals [!Ref SetUpLikePrd, "Y"] ThisIsELM: !Equals [!Ref UppercaseSFTPEnv, "ELM"] NoEips: !Equals [!Ref Eip1, "none"] YesEips: !Not [!Equals [!Ref Eip1, "none"]] OverridePath: !Equals [!Ref OverrideMountTarget, "Y"] OverrideEFSID: !Equals [!Ref OverrideEFS, "Y"] CreateSNSTopic: !Equals [!Ref SNSNotificationImport, ""] ImportSNSTopic: !Not [!Equals [!Ref SNSNotificationImport, ""]] SNSSubscribeEmail: !Not [!Equals [!Ref SNSNotificationEmail, ""]] #Mappings Section Mappings: EFSID: #nonprod account number "415418166582": "el": "fs-e15fa548" "hr": "fs-e62ef24f" "sa": "fs-a1a5c208" #prod account number "242275737326": "el": "fs-6e6facc7" "hr": "fs-e64d914f" "sa": "fs-a3117b0a"

Metadata

Metadata is mostly for organizing and presenting Parameters in a better way when using CloudFormation in the AWS Web UI.

Metadata: AWS::CloudFormation::Interface: ParameterGroups: - Label: default: Application Information Parameters: - UppercaseSFTPEnv - LowercasePillar - Label: default: Instance Settings Parameters: - SecurityGroupCloudFormationName - KeyName - EcsImageId - EcsInstanceType - SetUpLikePrd - Label: default: Load Balancer Settings Parameters: - Eip1 - Eip2 - LoggingLabel - Label: default: Application Settings Parameters: - DockerImage - OverrideMountTarget - OverrideMountPath - OverrideEFS - OverrideEFSVal - DRS3Bucket - SNSNotificationImport - SNSNotificationEmail - Label: default: Tags Parameters: - TagService - TagContactNetid - TagAccountNumber - TagSubAccount - TagTicketNumber

Resources

These are all of the actual AWS resources created for this application.

Resources:

Instance Role

This is the IAM role that will be applied to the ECS Instances. Any AWS specific permissions that the node might need should be defined here.

EnvInstanceRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: - ec2.amazonaws.com Action: - sts:AssumeRole Path: "/" ManagedPolicyArns: - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore Policies: - PolicyName: "dr-backup-access" PolicyDocument: Version: '2012-10-17' Statement: - Sid: Stmt1452033379000 Effect: Allow Action: - s3:ListBucket - s3:ListBucketVersions Resource: - !Sub "arn:aws:s3:::${DRS3Bucket}" - Sid: Stmt1452033379200 Effect: Allow Action: - s3:PutObject - s3:GetObject - s3:DeleteObject - s3:PutObjectAcl Resource: - !Sub "arn:aws:s3:::${DRS3Bucket}/*" - PolicyName: "sns-notifications" PolicyDocument: Version: '2012-10-17' Statement: - Sid: Stmt1452033379100 Effect: Allow Action: - sns:publish Resource: - !If - ImportSNSTopic - Fn::ImportValue: !Ref SNSNotificationImport - !Ref SFTPHubAlertTopic

Instance Profile

This is just a little construct to connect a set of roles together into a profile. The profile is referenced by ec2 instances.

EnvInstanceProfile: Type: AWS::IAM::InstanceProfile Properties: Path: "/" Roles: - !Ref EnvInstanceRole

Instance Security Group

Security group for the host nodes themselves. Needs to permit incoming traffice from the ELB, and any other authorized incoming sources.

InstanceSecurityGroup: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: Allow Load Balancer and SSH to host VpcId: Fn::ImportValue: !Sub "${SecurityGroupCloudFormationName}-VPCID" SecurityGroupIngress:

Opening up to all traffic as Network LBs don't allow SG on external side. This is normal with NLB, as security is enforced at the host.

- IpProtocol: tcp FromPort: 32768 ToPort: 51677 CidrIp: 0.0.0.0/0 Description: Dynamic port NLB 1 - IpProtocol: tcp FromPort: 51681 ToPort: 60999 CidrIp: 0.0.0.0/0 Description: Dynamic port NLB 2 - IpProtocol: tcp FromPort: 22 ToPort: 22 CidrIp: 10.138.2.0/24 Description: Mosaic VPN-1 - IpProtocol: tcp FromPort: 22 ToPort: 22 CidrIp: 150.135.241.0/24 Description: Mosaic VPN-2 - IpProtocol: tcp FromPort: 22 ToPort: 22 CidrIp: 150.135.112.0/24 Description: InfraDev VPN - IpProtocol: tcp FromPort: 22 ToPort: 22 CidrIp: 128.196.130.211/32 Description: ben.uits bastion Tags: - Key: service Value: !Ref TagService - Key: Name Value: !Sub "${LowercasePillar}-sftp-hub-inst-sg" - Key: environment Value: !Sub "${LowercasePillar}-sftp-hub" - Key: contactnetid Value: !Ref TagContactNetid - Key: accountnumber Value: !Ref TagAccountNumber - Key: ticketnumber Value: !Ref TagTicketNumber - Key: subaccount Value: !Ref TagSubAccount

Elastic Load Balancer (ELB)

Create this LB if we are using Elastic IPs so we can provide vendors with a set of static IPs

SftpNetworkLoadBalancerEip: Type: AWS::ElasticLoadBalancingV2::LoadBalancer Condition: YesEips Properties: Name: !Sub "peoplesoft-${LowercasePillar}-sftp-network-lb" Type: "network" Scheme: internet-facing ##Network Load Balancers don't support logging to S3 buckets #LoadBalancerAttributes: #- Key: access_logs.s3.enabled

Value: true

#- Key: access_logs.s3.bucket

Value: edu.arizona.iso.elb.logs

#- Key: access_logs.s3.prefix

Value: !Sub "ps${LowercasePillar}-${LoggingLabel}-sftp"

#Subnets:
  • Fn::ImportValue: !Sub "${SecurityGroupCloudFormationName}-PubSubNet1"
  • Fn::ImportValue: !Sub "${SecurityGroupCloudFormationName}-PubSubNet2"
SubnetMappings: - AllocationId: !Ref Eip1 SubnetId: Fn::ImportValue: !Sub "${SecurityGroupCloudFormationName}-PubSubNet1" - AllocationId: !Ref Eip2 SubnetId: Fn::ImportValue: !Sub "${SecurityGroupCloudFormationName}-PubSubNet2" Tags: - Key: service Value: !Ref TagService - Key: Name Value: !Sub "${LowercasePillar}-sftp-hub" - Key: environment Value: !Sub "${LowercasePillar}-sftp-hub" - Key: contactnetid Value: !Ref TagContactNetid - Key: accountnumber Value: !Ref TagAccountNumber - Key: ticketnumber Value: !Ref TagTicketNumber - Key: subaccount Value: !Ref TagSubAccount

Create this LB if we are NOT using Elastic IPs

SftpNetworkLoadBalancer: Type: AWS::ElasticLoadBalancingV2::LoadBalancer Condition: NoEips Properties: Name: !Sub "peoplesoft-${LowercasePillar}-sftp-network-lb" Type: "network" Scheme: internet-facing ##Network Load Balancers don't support logging to S3 buckets #LoadBalancerAttributes: #- Key: access_logs.s3.enabled

Value: true

#- Key: access_logs.s3.bucket

Value: edu.arizona.iso.elb.logs

#- Key: access_logs.s3.prefix

Value: !Sub "ps${LowercasePillar}-${LoggingLabel}-sftp"

Subnets: - Fn::ImportValue: !Sub "${SecurityGroupCloudFormationName}-PubSubNet1" - Fn::ImportValue: !Sub "${SecurityGroupCloudFormationName}-PubSubNet2" Tags: - Key: service Value: !Ref TagService - Key: Name Value: !Sub "${LowercasePillar}-sftp-hub" - Key: environment Value: !Sub "${LowercasePillar}-sftp-hub" - Key: contactnetid Value: !Ref TagContactNetid - Key: accountnumber Value: !Ref TagAccountNumber - Key: ticketnumber Value: !Ref TagTicketNumber - Key: subaccount Value: !Ref TagSubAccount #ELB Target group for SFTP ECS Cluster SftpELBV2Tg: Type: AWS::ElasticLoadBalancingV2::TargetGroup Properties: Name: !Sub "peoplesoft-${LowercasePillar}-sftp-tg" HealthCheckIntervalSeconds: 30 HealthCheckProtocol: TCP HealthCheckPort: traffic-port HealthCheckTimeoutSeconds: 10 HealthyThresholdCount: 2 UnhealthyThresholdCount: 2 Port: 2022 Protocol: TCP TargetGroupAttributes: - Key: "deregistration_delay.timeout_seconds" Value: "300" VpcId: Fn::ImportValue: !Sub "${SecurityGroupCloudFormationName}-VPCID" #ELB Listeners for SFTP Network LB SftpELBListener: Type: AWS::ElasticLoadBalancingV2::Listener Properties: DefaultActions: - Type: "forward" TargetGroupArn: !Ref SftpELBV2Tg LoadBalancerArn: !If [YesEips, !Ref SftpNetworkLoadBalancerEip, !Ref SftpNetworkLoadBalancer] Port: 22 Protocol: TCP

Route53 DNS Record

Create a DNS entry in Route53 for this environment. This creates a CNAME pointing at the DNS name of the Load Balancer.

AppDnsRecord: Type: AWS::Route53::RecordSet Properties:

Append a period after the hosted zone DNS name

HostedZoneName: Fn::Sub: - "${HostedZoneName}." - { HostedZoneName: !ImportValue peoplesoft-route53-dns } Name: Fn::Sub: - "sftp.${LowercasePillar}.${HostedZoneName}." - { HostedZoneName: !ImportValue peoplesoft-route53-dns } Type: CNAME TTL: 900 ResourceRecords: - !If [YesEips, !GetAtt SftpNetworkLoadBalancerEip.DNSName, !GetAtt SftpNetworkLoadBalancer.DNSName] #If this is ELM we need to create a legacy Route53 entry #sftp.peoplesoft-sftp-hub.(hosted zone) (i.e. sftp.peoplesoft-sftp-hub.ps-prod-aws.arizona.edu) LegacyDnsRecord: Type: AWS::Route53::RecordSet Condition: ThisIsELM Properties:

Append a period after the hosted zone DNS name

HostedZoneName: Fn::Sub: - "${HostedZoneName}." - { HostedZoneName: !ImportValue peoplesoft-route53-dns } Name: Fn::Sub: - "sftp.peoplesoft-sftp-hub.${HostedZoneName}." - { HostedZoneName: !ImportValue peoplesoft-route53-dns } Type: CNAME TTL: 900 ResourceRecords: - !If [YesEips, !GetAtt SftpNetworkLoadBalancerEip.DNSName, !GetAtt SftpNetworkLoadBalancer.DNSName] #Need to create a LogGroup in order for the ECS service to log details of the build #If this does not exist the ECS Service will not come up EcsLogGroup: Type: AWS::Logs::LogGroup Properties: LogGroupName: !Sub "${UppercaseSFTPEnv}-SFTP-Hub-LG" RetentionInDays: 30 #Launch Config for the Auto Scaling Group for the ECS Cluster EcsInstanceLc: Type: AWS::AutoScaling::LaunchConfiguration Properties: ImageId: !Ref EcsImageId InstanceType: !Ref EcsInstanceType AssociatePublicIpAddress: false IamInstanceProfile: !Ref EnvInstanceProfile KeyName: !Ref KeyName SecurityGroups: - !Ref InstanceSecurityGroup - Fn::ImportValue: !Sub "${SecurityGroupCloudFormationName}-ELMEFSSG" - Fn::ImportValue: !Sub "${SecurityGroupCloudFormationName}-HREFSSG" - Fn::ImportValue: !Sub "${SecurityGroupCloudFormationName}-SAEncryptedEFSSG" - Fn::ImportValue: !Sub "${SecurityGroupCloudFormationName}-SshSg" BlockDeviceMappings: - DeviceName: "/dev/xvda" Ebs: Encrypted: true VolumeSize: 30 VolumeType: gp2 UserData: Fn::Base64: !Sub | #!/bin/bash echo ECS_CLUSTER=${UppercaseSFTPEnv}SFTP >> /etc/ecs/ecs.config echo -ne "# SFTPHub container network optimizations\nnet.ipv4.tcp_tw_reuse = 1\nnet.ipv4.ip_forward = 1\n" >> /etc/sysctl.d/01-sftphub.conf sysctl --system #Auto Scaling Group for Web/App/Batch #Will be used in all non-prod environments, if this is prod this will not be created EcsInstanceAsg: Type: AWS::AutoScaling::AutoScalingGroup DependsOn: EcsCluster Properties: VPCZoneIdentifier: - Fn::ImportValue: !Sub "${SecurityGroupCloudFormationName}-PrivSubNet1" - Fn::ImportValue: !Sub "${SecurityGroupCloudFormationName}-PrivSubNet2" LaunchConfigurationName: !Ref EcsInstanceLc MinSize: '0' #If this is production we'll need to instances for HA, for nonprod only 1 MaxSize: !If [ThisIsProd,"4","2"] DesiredCapacity: !If [ThisIsProd,"2","1"] Tags: - Key: Name Value: !Sub "ECS Instance - ${AWS::StackName}" PropagateAtLaunch: true - Key: Description Value: "This instance is the part of the Auto Scaling group which was created through CloudFormation" PropagateAtLaunch: true - Key: service Value: !Ref TagService PropagateAtLaunch: true - Key: environment Value: !Sub "${LowercasePillar}-sftp-hub" PropagateAtLaunch: true - Key: contactnetid Value: !Ref TagContactNetid PropagateAtLaunch: true - Key: accountnumber Value: !Ref TagAccountNumber PropagateAtLaunch: true - Key: ticketnumber Value: !Ref TagTicketNumber PropagateAtLaunch: true - Key: subaccount Value: !Ref TagSubAccount PropagateAtLaunch: true #ECS Cluster EcsCluster: Type: AWS::ECS::Cluster #Need to make sure the LB is created before the ECS cluster is created Properties: ClusterName: !Sub "${UppercaseSFTPEnv}SFTP" #ECS Task Definition EcsTask: Type: AWS::ECS::TaskDefinition Properties: Family: !Sub "${UppercaseSFTPEnv}-SFTP" NetworkMode: "bridge" ContainerDefinitions: - Name: !Sub "${UppercaseSFTPEnv}-SFTP" Essential: true Image: !Ref DockerImage PortMappings: - HostPort: 0 ContainerPort: 22 Protocol: tcp Hostname: !Sub "${LowercasePillar}-sftp" Cpu: 200 MemoryReservation: 512 Privileged: true Environment: - Name: "SFTP_ENV" Value: !Sub "${UppercaseSFTPEnv}" - Name: "SFTP_MOUNTS" Value: Fn::Sub: - "${ThisEFSID}.efs.${AWS::Region}.amazonaws.com:/=${MountPath}" - ThisEFSID: Fn::If: - OverrideEFSID - !Ref OverrideEFSVal - Fn::FindInMap: [EFSID,!Ref "AWS::AccountId",!Ref LowercasePillar] MountPath: Fn::If: [OverridePath, !Ref OverrideMountPath, !Sub "/mosaic/${UppercaseSFTPEnv}"] - Name: "SNS_NOTIFICATION_TOPIC" Value: Fn::If: - ImportSNSTopic - Fn::ImportValue: !Ref SNSNotificationImport - !Ref SFTPHubAlertTopic - Name: "BackupSourcePath" Value: Fn::Sub: - "${MountPath}" - MountPath: Fn::If: [OverridePath, !Ref OverrideMountPath, !Sub "/mosaic/${UppercaseSFTPEnv}"] - Name: "S3DRBackupURI" Value: !Sub "s3://${DRS3Bucket}/${LowercasePillar}${LoggingLabel}" LogConfiguration: LogDriver: "awslogs" Options: awslogs-group: !Ref EcsLogGroup awslogs-region: !Ref AWS::Region awslogs-stream-prefix: "SFTP-HUB"

ECS Service

EcsService: Type: AWS::ECS::Service #Waiting for the DNS record to be created beause we know the LB has been #created if the DNS record is created, and the LB needs to be created #before the ECS service is created DependsOn: - EcsInstanceAsg - AppDnsRecord Properties: ServiceName: !Sub "${UppercaseSFTPEnv}-SFTP" Cluster: !Ref EcsCluster TaskDefinition: !Ref EcsTask DesiredCount: !If [ThisIsProd,2,1] HealthCheckGracePeriodSeconds: 60 Role: !Ref EcsServiceRole LoadBalancers: - ContainerName: !Sub "${UppercaseSFTPEnv}-SFTP" ContainerPort: 22 TargetGroupArn: !Ref SftpELBV2Tg #For now we will spread across AZs PlacementStrategies: - Field: "attribute:ecs.availability-zone" Type: "spread" #Will take the defaults for this right now, may not apply to PeopleSoft DeploymentConfiguration: MaximumPercent: 200 #PlacementConstraints:
  • PlacementConstraints, ...

ECS Service Role

EcsServiceRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: - ecs.amazonaws.com Action: - sts:AssumeRole Path: "/" ManagedPolicyArns: - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole

SNS Topic For Notifications

This creates an SNS topic which will receive notifications for the various alerts from the SFTPHub. An initial email address (passed in via parameters above) is set as a subscriber.

SFTPHubAlertTopic: Type: AWS::SNS::Topic Condition: CreateSNSTopic Properties: TopicName: !Sub "${AWS::StackName}-Alert" DisplayName: !Sub "SFTPHub ${AWS::StackName} Alert" Subscription: - !If - SNSSubscribeEmail - Endpoint: !Ref SNSNotificationEmail Protocol: email - !Ref "AWS::NoValue" Tags: - Key: Name Value: !Sub "${AWS::StackName} SNS Alert" - Key: service Value: !Ref TagService - Key: environment Value: !Sub "${LowercasePillar}-sftp-hub" - Key: contactnetid Value: !Ref TagContactNetid - Key: accountnumber Value: !Ref TagAccountNumber - Key: ticketnumber Value: !Ref TagTicketNumber - Key: subaccount Value: !Ref TagSubAccount

Outputs

Output values that can be viewed from the AWS CloudFormation console.

Outputs: LoadBalancerDNS: Value: !If [YesEips, !GetAtt SftpNetworkLoadBalancerEip.DNSName, !GetAtt SftpNetworkLoadBalancer.DNSName] Export: Name: !Sub "${AWS::StackName}-lb-dns" SftpEndpointDNS: Value: !Ref AppDnsRecord Export: Name: !Sub "${AWS::StackName}-dns" SftpSNSTopic: Value: !If - CreateSNSTopic - !Ref SFTPHubAlertTopic - !Ref "AWS::NoValue" Export: Name: !Sub "${AWS::StackName}-topic"