kuali_sonarqube_ecs.yaml
---

Kuali SonarQube CloudFormation Deployment

This CloudFormation template will build an ECS stack to support the Kuali team's SonarQube instance that is backed by EFS for the related data storage.

AWSTemplateFormatVersion: '2010-09-09' Description: Kuali SonarQube ECS

Parameters

These are the input parameters for this template. All of these parameters must be supplied for this template to be deployed.

Parameters: AppSlug: MinLength: '3' Type: String Description: Short application slug, ie 'kfs'. Lowercase letters, numbers and dashes only AllowedPattern: "[a-z0-9-]*" KeyName: Description: Amazon EC2 Key Pair Type: AWS::EC2::KeyPair::KeyName Default: "kfs-development-environments-keypair" DockerImage: Description: 'Docker Image, i.e.: kuali/sonarqube:kuali-sonarqube-X.x-community-DATE-BUILD_NUMBER' Default: 397167497055.dkr.ecr.us-west-2.amazonaws.com/kuali/sonarqube:kuali-sonarqube-7.7-community-2019-05-31-1 Type: String EcsImageId: Description: The AMI Amazon built specifically for ECS Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id> Default: /aws/service/ecs/optimized-ami/amazon-linux/recommended/image_id EcsInstanceType: Description: ECS Instance AWS Server Type Type: String Default: "t3.medium"

To address in the future for DR purposes. DRS3Bucket: Description: 'Disaster Recovery Bucket Name' Default: "edu-arizona-dr-kuali" Type: String

EFSStackName: MinLength: '2' Type: String Description: Name of the EFS CloudFormation Stack Default: kuali-sonarqube-efs HostedZoneName: MinLength: '3' Type: String Description: 'Name of Route53 Hosted Zone: ie ''aws.arizona.edu''' Default: "ua-uits-kuali-nonprod.arizona.edu" SSLCertARN: Description: Application SSL Certificate ARN Type: String Default: "arn:aws:acm:us-west-2:397167497055:certificate/9a4ee0ac-1031-41c5-9457-0181eab28f7b" JDBCUserName: Description: Username for jdbc database used to connect SonarQube Type: String JDBCPassword: Description: URL for jdbc database used to connect SonarQube Type: String JDBCURL: Description: URL for jdbc database used to connect SonarQube Type: String EfsDirectoryInstance: Description: An existing EFS directory to be used for the mount. Type: String TagService: Description: Refers to the application (Uaccess Learning, Uaccess Employee, Uaccess Student) Type: String Default: "Uaccess Financials" TagApplication: Description: The specific application of this resource Type: String Default: "Kuali SonarQube Static Analysis" TagEnvironment: Description: Type of environment that is using this resource, such as 'dev', 'tst', 'prd'. Type: String Default: "dev" TagContactNetid: Description: NetID of person most familiar with resource Type: String Default: "fimbresrc" TagAccountNumber: Description: Identifies the financial system account number Type: String Default: "1192620" TagSubAccount: Description: Financial system subaccount number for the service utilizing this resource Type: String Default: "12AWS" TagTicketNumber: Description: Jira Ticket Number Type: String Default: "FIN-920"

Metadata

Metadata is mostly for organizing and presenting Parameters in a better way when using CloudFormation in the AWS Web UI.

Metadata: AWS::CloudFormation::Interface: ParameterGroups: - Label: default: Application Information Parameters: - AppSlug - Label: default: Instance Settings Parameters: - KeyName - EcsImageId - EcsInstanceType - Label: default: Application Settings Parameters: - DockerImage - EFSStackName - Label: default: Tags Parameters: - TagService - TagApplication - TagName - TagEnvironment - TagContactNetid - TagAccountNumber - TagSubAccount - TagTicketNumber

These are all of the actual AWS resources created for this application.

Resources:

Instance Role

This is the IAM role that will be applied to the ECS Instances. Any AWS specific permissions that the node might need should be defined here.

EnvInstanceRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: - ec2.amazonaws.com Action: - sts:AssumeRole Path: "/" ManagedPolicyArns: - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role - arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM
 Policies:
 -
   PolicyName: "dr-backup-access"
   PolicyDocument:
     Version: '2012-10-17'
     Statement:
     -
       Sid: Stmt1452033379000
       Effect: Allow
       Action:
       - s3:ListBucket
       - s3:PutObject
       Resource:
       - !Sub "arn:aws:s3:::${DRS3Bucket}"
       - !Sub "arn:aws:s3:::${DRS3Bucket}/*"

Instance Profile

This is just a little construct to connect a set of roles together into a profile. The profile is referenced by ec2 instances.

EnvInstanceProfile: Type: AWS::IAM::InstanceProfile Properties: Path: "/" Roles: - !Ref EnvInstanceRole

Instance Security Group

Security group for the host nodes themselves. Needs to permit incoming traffice from the ELB, and any other authorized incoming sources.

InstanceSecurityGroup: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: Allow Load Balancer and SSH to host VpcId: !ImportValue kuali-vpc-vpcid SecurityGroupIngress: - IpProtocol: "tcp" FromPort: "31000" ToPort: "61000" SourceSecurityGroupId: !Ref 'EcsAlbSecurityGroup' Description: Allow incoming traffic from the load balancer - IpProtocol: tcp FromPort: '22' ToPort: '22' CidrIp: 10.138.2.0/24 Description: Mosaic VPN-1 - IpProtocol: tcp FromPort: '22' ToPort: '22' CidrIp: 150.135.241.0/24 Description: Mosaic VPN-2 - IpProtocol: tcp FromPort: '22' ToPort: '22' CidrIp: 150.135.112.0/24 Description: InfraDev VPN - IpProtocol: tcp FromPort: '22' ToPort: '22' CidrIp: 128.196.130.211/32 Description: ben.uits bastion Tags: - Key: service Value: !Ref TagService - Key: application Value: !Ref TagApplication - Key: Name Value: !Sub "${AppSlug}-sonarqube-efs-inst-sg" - Key: environment Value: !Ref TagEnvironment - Key: contactnetid Value: !Ref TagContactNetid - Key: accountnumber Value: !Ref TagAccountNumber - Key: subaccount Value: !Ref TagSubAccount - Key: ticketnumber Value: !Ref TagTicketNumber

Application Load Balancer (ALB)

Defines the Application Load Balancer Reference: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-elasticloadbalancingv2-loadbalancer.html

SonarQubeApplicationLoadBalancer: Type: "AWS::ElasticLoadBalancingV2::LoadBalancer" Properties:
Name: !Sub "${AppSlug}-sonarqube-alb" Scheme: internal Subnets: - !ImportValue kuali-vpc-private-subnet-a - !ImportValue kuali-vpc-private-subnet-b LoadBalancerAttributes: - Key: idle_timeout.timeout_seconds Value: 60 SecurityGroups: - !Ref EcsAlbSecurityGroup Type: application IpAddressType: ipv4 Tags: - Key: service Value: !Ref TagService - Key: application Value: !Ref TagApplication - Key: Name Value: !Sub "${AppSlug}-sonarqube-alb" - Key: environment Value: !Ref TagEnvironment - Key: contactnetid Value: !Ref TagContactNetid - Key: accountnumber Value: !Ref TagAccountNumber - Key: subaccount Value: !Ref TagSubAccount - Key: ticketnumber Value: !Ref TagTicketNumber EcsAlbSecurityGroup: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: 'Allow external traffic to Kuali SonarQube load balancer' VpcId: !ImportValue kuali-vpc-vpcid SecurityGroupIngress: - IpProtocol: tcp FromPort: '80' ToPort: '80' CidrIp: 10.138.2.0/24 Description: Mosaic VPN-1 - IpProtocol: tcp FromPort: '80' ToPort: '80' CidrIp: 150.135.241.0/24 Description: Mosaic VPN-2 - IpProtocol: tcp FromPort: '80' ToPort: '80' CidrIp: 150.135.112.0/24 Description: InfraDev VPN - IpProtocol: tcp FromPort: '443' ToPort: '443' CidrIp: 10.138.2.0/24 Description: Mosaic VPN-1 - IpProtocol: tcp FromPort: '443' ToPort: '443' CidrIp: 150.135.241.0/24 Description: Mosaic VPN-2 - IpProtocol: tcp FromPort: '443' ToPort: '443' CidrIp: 150.135.112.0/24 Description: InfraDev VPN - IpProtocol: tcp FromPort: '443' ToPort: '443' CidrIp: 10.220.176.0/23 Description: Local VPC traffic Tags: - Key: "Name" Value: !Sub "${AppSlug}-sonarqube-alb-sg" - Key: service Value: !Ref TagService - Key: application Value: !Ref TagApplication - Key: environment Value: !Ref TagEnvironment - Key: contactnetid Value: !Ref TagContactNetid - Key: accountnumber Value: !Ref TagAccountNumber - Key: subaccount Value: !Ref TagSubAccount - Key: ticketnumber Value: !Ref TagTicketNumber

DB Security Group

Defines the Security Group for the RDS Database. This restricts DB access to only the devices in the InstanceSecurityGroup, so our App nodes.

DBSecurityGroup: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: Allow DB traffic from Application Instances VpcId: !ImportValue kuali-vpc-vpcid SecurityGroupIngress: - IpProtocol: tcp FromPort: '1521' ToPort: '1521' SourceSecurityGroupId: !Ref InstanceSecurityGroup Tags: - Key: service Value: !Ref TagService - Key: Name Value: !Sub "sonar-db-sg" - Key: environment Value: !Ref TagEnvironment - Key: createdby Value: !Ref TagContactNetid - Key: contactnetid Value: !Ref TagContactNetid - Key: accountnumber Value: !Ref TagAccountNumber - Key: subaccount Value: !Ref TagSubAccount - Key: ticketnumber Value: !Ref TagTicketNumber

ELB Target group for SonarQube ECS Cluster

SonarQubeELBV2Tg: Type: "AWS::ElasticLoadBalancingV2::TargetGroup" Properties: Name: !Sub "${AppSlug}-sonarqube-tg" HealthCheckIntervalSeconds: 30 HealthCheckPath: / HealthCheckTimeoutSeconds: 10 HealthyThresholdCount: 2 Matcher: HttpCode: "200-399" UnhealthyThresholdCount: 2 Port: 80 Protocol: HTTP TargetGroupAttributes: - Key: "deregistration_delay.timeout_seconds" Value: "300" VpcId: !ImportValue kuali-vpc-vpcid Tags: - Key: "Name" Value: !Sub "${AppSlug}-sonarqube-tg" - Key: service Value: !Ref TagService - Key: application Value: !Ref TagApplication - Key: environment Value: !Ref TagEnvironment - Key: contactnetid Value: !Ref TagContactNetid - Key: accountnumber Value: !Ref TagAccountNumber - Key: subaccount Value: !Ref TagSubAccount - Key: ticketnumber Value: !Ref TagTicketNumber

ELB Listeners for SonarQube Application LB

SonarQubeELBListener: Type: "AWS::ElasticLoadBalancingV2::Listener" Properties: Certificates: - CertificateArn: !Ref SSLCertARN DefaultActions: - Type: "forward" TargetGroupArn: !Ref "SonarQubeELBV2Tg" LoadBalancerArn: !Ref SonarQubeApplicationLoadBalancer Port: "443" Protocol: "HTTPS" SslPolicy: "ELBSecurityPolicy-TLS-1-1-2017-01"

Route53 DNS Record

Create a DNS entry in Route53 for this environment. This creates a CNAME pointing at the DNS name of the Load Balancer.

AppDnsRecord: Type: AWS::Route53::RecordSet Properties:

Append a period after the hosted zone DNS name

HostedZoneName: !Sub "${HostedZoneName}." Name: !Sub "${AppSlug}.${HostedZoneName}." Type: CNAME TTL: '900' ResourceRecords: - !GetAtt SonarQubeApplicationLoadBalancer.DNSName

Need to create a LogGroup in order for the ECS service to log details of the build If this does not exist the ECS Service will not come up

EcsLogGroup: Type: AWS::Logs::LogGroup Properties: LogGroupName: !Sub "${AppSlug}-sonarqube-lg" RetentionInDays: 30

Launch Config for the Auto Scaling Group for the ECS Cluster

EcsInstanceLc: Type: AWS::AutoScaling::LaunchConfiguration Properties: ImageId: !Ref EcsImageId InstanceType: !Ref EcsInstanceType InstanceMonitoring: false AssociatePublicIpAddress: false IamInstanceProfile: !Ref EnvInstanceProfile KeyName: !Ref KeyName SecurityGroups: - !Ref InstanceSecurityGroup - Fn::ImportValue: !Sub "${EFSStackName}-target-sg" BlockDeviceMappings: - DeviceName: "/dev/xvdcz" Ebs: VolumeSize: "22" VolumeType: "gp2" UserData: Fn::Base64: !Sub - | Content-Type: multipart/mixed; boundary="==BOUNDARY==" MIME-Version: 1.0 --==BOUNDARY== Content-Type: text/cloud-boothook; charset="us-ascii"

Install nfs-utils

cloud-init-per once yum_update yum update -y cloud-init-per once install_nfs_utils yum install -y nfs-utils

Create /efs folder

cloud-init-per once mkdir_efs mkdir -p "/efs"

Mount /efs

cloud-init-per once mount_efs echo -e "${efsid}.efs.us-west-2.amazonaws.com:/ /efs nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 0 0" >> /etc/fstab mount -a --==BOUNDARY== Content-Type: text/x-shellscript; charset="us-ascii"

create the dirs on the efs mount that will be used to persist data.

mkdir -p "/efs/${EfsDirectoryInstance}/data" mkdir -p "/efs/${EfsDirectoryInstance}/conf" mkdir -p "/efs/${EfsDirectoryInstance}/logs" sudo chmod -R 777 /efs #!/bin/bash

Set any ECS agent configuration options

echo ECS_CLUSTER=${AppSlug}-SONARQUBE >> /etc/ecs/ecs.config --==BOUNDARY==-- - efsid: Fn::ImportValue: !Sub "${EFSStackName}-fs-id"

Auto Scaling Group for Web/App/Batch Will be used in current non-prod environment

EcsInstanceAsg: Type: AWS::AutoScaling::AutoScalingGroup DependsOn: EcsCluster Properties: VPCZoneIdentifier: - !ImportValue kuali-vpc-private-subnet-a - !ImportValue kuali-vpc-private-subnet-b LaunchConfigurationName: !Ref EcsInstanceLc MinSize: '0' MaxSize: '1' DesiredCapacity: '1' TargetGroupARNs: - !Ref SonarQubeELBV2Tg Tags: - Key: Name Value: !Sub "${AWS::StackName} ECS Host" PropagateAtLaunch: 'true' - Key: Description Value: "This instance is the part of the Auto Scaling group which was created through CloudFormation" PropagateAtLaunch: 'true' - Key: service Value: !Ref TagService PropagateAtLaunch: 'true' - Key: application Value: !Ref TagApplication PropagateAtLaunch: 'true' - Key: environment Value: !Ref TagEnvironment PropagateAtLaunch: 'true' - Key: contactnetid Value: !Ref TagContactNetid PropagateAtLaunch: 'true' - Key: accountnumber Value: !Ref TagAccountNumber PropagateAtLaunch: 'true' - Key: subaccount Value: !Ref TagSubAccount PropagateAtLaunch: 'true' - Key: ticketnumber Value: !Ref "TagTicketNumber" PropagateAtLaunch: 'true'

ECS Cluster

EcsCluster: Type: "AWS::ECS::Cluster" #Need to make sure the LB is created before the ECS cluster is created Properties: ClusterName: !Sub "${AppSlug}-SONARQUBE"

ECS Task Definition

EcsTask: Type: "AWS::ECS::TaskDefinition" Properties: Family: !Sub "${AppSlug}-SONARQUBE" NetworkMode: "bridge"
ContainerDefinitions: - Name: !Sub "${AppSlug}-SONARQUBE" Essential: "true" Image: !Ref DockerImage PortMappings: - HostPort: "0" ContainerPort: "9000" Protocol: "tcp" Hostname: !Sub "${AppSlug}-sonarqube" Cpu: "400" MemoryReservation: "1024" Privileged: "true" Environment: - Name: "sonar.jdbc.username" Value: !Ref JDBCUserName - Name: "sonar.jdbc.password" Value: !Ref JDBCPassword - Name: "sonar.jdbc.url" Value: !Ref JDBCURL LogConfiguration: LogDriver: "awslogs" Options: awslogs-group: !Ref "EcsLogGroup" awslogs-region: !Ref "AWS::Region" awslogs-stream-prefix: "KUALI-SONARQUBE" MountPoints: - ContainerPath: "/opt/sonarqube/extensions" SourceVolume: "efs-sonar-ext" - ContainerPath: "/opt/sonarqube/data" SourceVolume: "efs-sonar-data" - ContainerPath: "/opt/sonarqube/conf" SourceVolume: "efs-sonar-conf" - ContainerPath: "/opt/sonarqube/logs" SourceVolume: "efs-sonar-logs"

Persistent storage for SonarQube handled via Docker volumes and bind mounts. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_data_volumes.html

Volumes: - Name: "efs-sonar-ext"

Using a DockerVolumeConfiguration property along with the MountPoints property from above will allow for using Docker volumes.
See https://docs.aws.amazon.com/AmazonECS/latest/developerguide/docker-volumes.html

Docker will map volume to default location on EC2 host at /var/lib/docker/volumes/efs-sonar-ext/_data.
If docker container is restarted then the "extensions" directory will persist.
If the EC2 host is restarted then the "extensions" dir will NOT persist.

DockerVolumeConfiguration: Scope: "shared" Autoprovision: "true" Driver: "local" - Name: "efs-sonar-data"

Using a Host property along with the MountPoints property from above will allow for using bind mounts. See https://docs.aws.amazon.com/AmazonECS/latest/developerguide/bind-mounts.html

Mapping explictly set using SourcePath property on EC2 host mapped to EFS. If docker container is restarted then the "extensions" directory will persist.
If the EC2 host is restarted then the SourcePath location dir will also persist.

Host: SourcePath: !Sub "/efs/${EfsDirectoryInstance}/data" - Name: "efs-sonar-conf" Host: SourcePath: !Sub "/efs/${EfsDirectoryInstance}/conf" - Name: "efs-sonar-logs" Host: SourcePath: !Sub "/efs/${EfsDirectoryInstance}/logs"

ECS Service

EcsService: Type: "AWS::ECS::Service" #Waiting for the DNS record to be created beause we know the LB has been #created if the DNS record is created, and the LB needs to be created #before the ECS service is created DependsOn: - EcsInstanceAsg - AppDnsRecord Properties: ServiceName: !Sub "${AppSlug}-SONARQUBE" Cluster: !Ref EcsCluster TaskDefinition: !Ref EcsTask DesiredCount: "1" Role: !Ref EcsServiceRole LoadBalancers: - ContainerName: !Sub "${AppSlug}-SONARQUBE" ContainerPort: "9000" TargetGroupArn: !Ref SonarQubeELBV2Tg #For now we will spread across AZs PlacementStrategies: - Field: "attribute:ecs.availability-zone" Type: "spread" DeploymentConfiguration: MaximumPercent: "100" MinimumHealthyPercent: "0"

ECS Service Role

EcsServiceRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: - ecs.amazonaws.com Action: - sts:AssumeRole Path: "/" ManagedPolicyArns: - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole

Outputs

Output values that can be viewed from the AWS CloudFormation console.

Outputs: LoadBalancerDNS: Value: !GetAtt SonarQubeApplicationLoadBalancer.DNSName Export: Name: !Sub "${AWS::StackName}-lb-dns" SonarQubeEndpointDNS: Value: !Ref AppDnsRecord Export: Name: !Sub "${AWS::StackName}-dns" SonarQubeURL: Value: !Sub "https://${AppDnsRecord}/" Export: Name: !Sub "${AWS::StackName}-url" SonarQubeEcsCluster: Value: !Ref EcsCluster Export: Name: !Sub "${AWS::StackName}-ecscluster" SonarQubeEcsClusterArn: Value: !GetAtt EcsCluster.Arn Export: Name: !Sub "${AWS::StackName}-ecsclusterarn"