One of my team members is retiring, and as we were talking about their tenure with the company, “We did a LOT!” they said. That got me looking back at the past 2 years, and I started listing some of the changes and accomplishments that happened, first while I was leading the engineering team as a whole, then as I shifted focus to lead DevOps and Information Security (and data engineering).
- Launched a new website on an aggressive, “impossible to do” timeline
- Rewrote user authentication framework
- Reengineered and stabilized flagship revenue application (2019 edition)
- Restructured cloud network to improve stability (2019 edition)
- Reengineered and stabilized flagship revenue application (2020 edition)
- Rearchitected tight coupling between cloud and on-prem systems to improve stability and reliability
- Grew in-house services dashboard to detect service interconnection issues
- Successfully upgraded to MongoDB v3.6
- Drove adoption of Eslint for better code quality
- Initiated annual town halls to establish mission and vision
- Initiated quarterly all-hands engineering meetings to acknowledge team efforts
- Improved code quality as measured by number of bugs logged
- Initiated a Tech Leads round table to share what is working not working, cross-pollinate ideas across projects
- Initiated Brown Bag lunches to increase knowledge and stimulate ideas to the benefit of the company
- Created application security engineer role
- Implemented static code analysis with Sonarqube to improve code security
- Initiated content security policy for website to improve security
- Upgraded all websites to TLS v1.2 to improve security
- Increased code testing coverage by double digits to over 50% in all projects and services
- Implemented client telemetry to detect user issues before calls come into help desk
- Consolidated workloads into one cloud, saving $10K/month
- Cut monthly AWS costs by double digits
- Separated AWS usage into multiple accounts by usage type, thereby increasing security
- Implemented multifactor authentication in AWS, MongoDB Atlas, and all other Saas platforms
- Reduced DNS footprint by about 40%, thereby reducing attack surface
- Cut all on-prem usage and migrated to be fully in AWS
- Stabilized in-house CICD tool so that builds fail far less frequently
- Successfully upgraded PHP from v5.6 to v7.3
- Successfully Updated to MongoDB v4.2
- Implemented Site-to-site VPN between AWS and on-prem
- Set up AWS Client VPN with OpenLDAP/SAML and Active Directory with MFA auth
- Migrated workloads from one Kubernetes clusters to another
- Upgraded Kubernetes versions
- Moved all our workloads from open subnets and publicly exposed EC2 instances to private subnets and non-public instances
- Separated cloud traffic into internal and external load balancers
- Set up internal Route53
- Created tool to quickly (within minutes) set up autonomous and fully equipped test environment with starter data
- Created 30 test environments so far (dev, qa, uat, stg) x 7 projects
- Moved Java-based application with clunky 1-hour deployment process to 5-minute Kubernetes deployment
- Moved EC2-attached NFS storage to serverless EFS for Kubernetes workloads
- Prepared static websites for serverless deployment using Amplify
- Set up Kubernetes observability tools
- Optimized MongoDB queries to remove spikes and smooth out CPU usage
- Set up secure FTP server with FUSE-based mount to S3 as storage
- Aggregate selected syslogs to centralized log server
- Reduced non-essential log entries by 90% (millions to thousands) to make errors easier to find
- Configured Systems Manger to manage EC2 fleet
- Set up AWS DataSync to move data between S3 and EFS
- Created AWS Lambda functions to automate DevOps tasks
- Configured Kubernetes ephemeral storage on EFS volumes to prevent disk pressure issues
That’s a good place to stop. All this could not have been accomplished without the dedicated team of engineers, PMs and QA teams, so it’s very much a collective effort.
I am looking forward to what’s coming up, such as more gitops, DevSecOps, infra as code, dynamic analysis (DAST), to name just a few.