Apple, Inc. - Cupertino, CA
[ Jun 2018 - Present ]
Cloud Services Systems SRE - Manage a fleet of servers which underpin much of Apple's internet-facing services
- Technical leadership role, guide monitoring, alerting, DNS, DHCP, and other infrastructure services. Cross-train SRE teams to troubleshoot network issues, escalate with partner teams, and engage with network teams
- Early in the role, worked to improve on-call quality-of-life by encouraging the team to correct alerts, add telemetry, and repair toil. This reduced our on-call incidents from five/week to once/quarter
- Work through incidents, resolve outages, perform root-cause analysis, document post-mortems, identify fixes, measure solutions, and finally deploy them
- Drive adoption of IPv6 in our Cloud Services Kubernetes platform. Hosts, platform, and workloads dual-stacked from the start, allowing us to easily provide IPv6-enabled services to our customers
- Worked with our Kubernetes team to support multiple workload IPv4 address classes, easing adoption of additional address classes without additional effort. This enabled us to reduce address demand considerably
- Push adoption of IPv6 in company-wide Apple services, like DNS, NTP, mail, authentication, user networks, and VPNs. One example, an eighteen month cross-functional project to add dual-stack support for NTP in macOS, iOS, and time.apple.com. Worked with MacOS and iOS teams to enable the feature. Worked with network teams to get policies deployed. Worked with the NTP team to enable the service. Worldwide go-live July 2021 without an issue
- Deployed our first IPv6-only Kubernetes clusters, worked with partner teams to enable IPv6-only supporting services as well. Included support for dual-stacked workloads, which meant configuring hosts for RFC5549
[ Nov 2014 - Jun 2018 ]
Network Engineer - Design, deploy, and operate very large-scale infrastructure supporting millions of users around the world and growing very rapidly
- Designed Clos network fabrics including rack plans, cable plans, BOMs, routing, and addressing
- Worked on automated deployment tools for compute and network capacity, shortening installation times
- Integrated pieces of network configuration management into our server configuration management, to get efficiencies with ancillary services like provisionoing, authentication, and telemetry
- Troubleshot complex infrastructure and application level issues
- Wrote documentation to assist team members for future troubleshooting and external group knowledge transfer
- Planned our IPv6 strategy with our server team counterparts, updated routing policies and configurations to support this effort
Quicken Loans - Detroit, MI
[ Jul 2013 - Nov 2014 ]
Network Engineer - Responsible for internet presence with enterprise campus and WAN in a highly agile environment
- Work with our platform engineering teams to move certain parts of our front end to Amazon AWS
- Datacenter upgrades from C6500s to Nexus 7k/5k/2k, double-VPC between distribution and access, single-VPC to servers
- Firewall upgrade and policy migration from FWSMs to ASA 5585s
- Host MHacks.org hackathon, providing 800 people with reliable, usable, and isolated internet access for a weekend. 1200 devices, 450mbps sustained throughput on 120 WAPs
- Engineered a VDI POC with Nexus 5500s and Dell M1000E/FEX architecture. VPC for Ethernet, FC storage to FCOE servers
- Day to day operations including firewall ACLs, switch port add/changes, diagnose wired/wireless/WAN performance issues, administer monitoring systems
Member Driven Technologies - Warren, MI
[ Oct 2012 - Jul 2013 ]
Network Engineer - Manage a Cisco network in a regulated service provider environment
- Manage MPLS, Metro-E, and other WAN circuits
- Day to day operations include update firewall policies on multi-context ASAs and FWSM devices, DMVPN, tunnelcrypt VPNs
- Route redistribution and traffic engineering for remote branches. Clean up inter-datacenter routing
- Telemetry projects for syslog and netflow
Greektown Casino-Hotel - Detroit, MI
[ Jun 2009 - Oct 2012 ]
Network Engineer - Manage a Cisco network, administer primarily a Windows and VMware ESX environment
- Manage Cisco WAPs and WLCs. WiFi performance coverage and monitoring for expanded VoIP/VoWiFi use
- Work with vendors to diagnose network application problems using RSPANs, packet captures, port scans, and log analysis
- Project to replace core network equipment in a 24x7 operation, downtime less than 15 seconds
- Finalized Multicast Routing design for digital signage distribution, IPTV, and surveillance
- Standardize system monitoring for host, service, and disk space availability. Added switch uplink state monitoring, IDF/MDF environmental monitoring, and responsibility-based alert structure
- Implemented an RSA enVision SIEM to assist with Gaming, GLBA, PCI, and SOX regulation audits and compliance
- Designed and implemented redundant, scalable, and secure ethernet connectivity for 2600 slot machines. 420 access switches, 8 aggregation switches with 10g port-channels to distribution
- Day to day tasks include managing firewall policies, VPN policies, deploying equipment to support new construction, and managing maintenance contracts
Netlink - Madison Heights, MI
[ Jun 2007 - May 2009 ]
System Administrator - Manage the internal Windows and virtualization infrastructure. Provide support to small/medium business customers
- Change switch port VLANs for servers and workstations as needed
- Install and configure server virtualization with fibrechannel storage
- Consolidate and support two AD forests, virtualizing dozens of application and file servers, and streamline DNS
- Set up system telemetry. Cacti for time-series, nagios for availability monitoring and alerting
- Work with customers to install, manage, and migrate their database, mail, and web servers
- Mail gateway project, cleaning up MX records and migrate from Barracuda to ProofPoint mail gateways, which processes 80,000 messages per hour
Oakland University - Rochester, MI
- Bachelor Science in Computer Science in December 2012
Oakland Community College - Bloomfield Hills, MI
- Associates Degree of Computer Information Systems in June 2005
Skill Summary:
- Cloud Services: Amazon AWS (VPC default, Route53, EBS, RDS, S3, and VPN), Microsoft Azure (topology, VPN)
- Languages: Some golang, python, ruby, shell scripting
- Operating Systems: Linux, (CentOS, Debian/Ubuntu), Windows Desktops/Servers
- Switches/Routers: Broadcom ICOS, Cisco IOS, Cisco NXOS, Cumulus
- Monitoring: Cacti, MRTG, Nagios, Prometheus, SNMP
- Databases: MySQL, MS SQL Server
- Misc. Software: Apache, Bind, ISC-DHCP, OpenSSH, Puppet, tcpdump+Wireshark
Independent Projects:
MPCon - http://www.mpcon.org/
[ March 2004 - Nov 2014 ]
Coordinator and Infrastructure - MPCon has been an established LAN Party in south-eastern Michigan for 10 years. In that time, we have grown from a small gathering of QuakeWorld Team Fortress clan mates to the largest LAN party in Michigan, and one of the most sought after LAN parties in the Midwest - LANParty.com. Our last event drew over 500 attendees.
- Manage a dozen volunteers to plan, market, set up, operate, and tear down an event for 500 people over four days
- Plan and coordinate with facilities for cooling, internet access, power, and tables
- Configure and operate DHCP, DNS, internet access, intrusion detection, game servers, and LAN party management server
- Monitor attendee check-ins, seating, security
- Investigate power and network connectivity/performance issues
- Resolve hacks, intrusions, and viruses on the MPCon network