Jul 2021


One of my team members is retiring, and as we were talking about their tenure with the company, “We did a LOT!” they said. That got me looking back at the past 2 years, and I started listing some of the changes and accomplishments that happened, first while I was leading the engineering team as a whole, then as I shifted focus to lead DevOps and Information Security (and data engineering).

  1. Launched a new website on an aggressive, “impossible to do” timeline
  2. Rewrote user authentication framework
  3. Reengineered and stabilized flagship revenue application (2019 edition)
  4. Restructured cloud network to improve stability (2019 edition)
  5. Reengineered and stabilized flagship revenue application (2020 edition)
  6. Rearchitected tight coupling between cloud and on-prem systems to improve stability and reliability
  7. Grew in-house services dashboard to detect service interconnection issues
  8. Successfully upgraded to MongoDB v3.6
  9. Drove adoption of Eslint for better code quality
  10. Initiated annual town halls to establish mission and vision
  11. Initiated quarterly all-hands engineering meetings to acknowledge team efforts
  12. Improved code quality as measured by number of bugs logged
  13. Initiated a Tech Leads round table to share what is working not working, cross-pollinate ideas across projects
  14. Initiated Brown Bag lunches to increase knowledge and stimulate ideas to the benefit of the company
  15. Created application security engineer role
  16. Implemented static code analysis with Sonarqube to improve code security
  17. Initiated content security policy for website to improve security
  18. Upgraded all websites to TLS v1.2 to improve security
  19. Increased code testing coverage by double digits to over 50% in all projects and services
  20. Implemented client telemetry to detect user issues before calls come into help desk
  21. Consolidated workloads into one cloud, saving $10K/month
  22. Cut monthly AWS costs by double digits
  23. Separated AWS usage into multiple accounts by usage type, thereby increasing security
  24. Implemented multifactor authentication in AWS, MongoDB Atlas, and all other Saas platforms
  25. Reduced DNS footprint by about 40%, thereby reducing attack surface
  26. Cut all on-prem usage and migrated to be fully in AWS
  27. Stabilized in-house CICD tool so that builds fail far less frequently
  28. Successfully upgraded PHP from v5.6 to v7.3
  29. Successfully Updated to MongoDB v4.2
  30. Implemented Site-to-site VPN between AWS and on-prem
  31. Set up AWS Client VPN with OpenLDAP/SAML and Active Directory with MFA auth
  32. Migrated workloads from one Kubernetes clusters to another
  33. Upgraded Kubernetes versions
  34. Moved all our workloads from open subnets and publicly exposed EC2 instances to private subnets and non-public instances
  35. Separated cloud traffic into internal and external load balancers
  36. Set up internal Route53
  37. Created tool to quickly (within minutes) set up autonomous and fully equipped test environment with starter data
  38. Created 30 test environments so far (dev, qa, uat, stg) x 7 projects
  39. Moved Java-based application with clunky 1-hour deployment process to 5-minute Kubernetes deployment
  40. Moved EC2-attached NFS storage to serverless EFS for Kubernetes workloads
  41. Prepared static websites for serverless deployment using Amplify
  42. Set up Kubernetes observability tools
  43. Optimized MongoDB queries to remove spikes and smooth out CPU usage
  44. Set up secure FTP server with FUSE-based mount to S3 as storage
  45. Aggregate selected syslogs to centralized log server
  46. Reduced non-essential log entries by 90% (millions to thousands) to make errors easier to find
  47. Configured Systems Manger to manage EC2 fleet
  48. Set up AWS DataSync to move data between S3 and EFS
  49. Created AWS Lambda functions to automate DevOps tasks
  50. Configured Kubernetes ephemeral storage on EFS volumes to prevent disk pressure issues

That’s a good place to stop. All this could not have been accomplished without the dedicated team of engineers, PMs and QA teams, so it’s very much a collective effort.

I am looking forward to what’s coming up, such as more gitops, DevSecOps, infra as code, dynamic analysis (DAST), to name just a few.