POP: PMACS Operations Portal Kevin Lux luxk@upenn.edu @luxk on Slack https://kevinlux.info/

Talk Overview Background Problem statement and POP design POP integrations / sample workflows Demo Conclusion and Q/A

Background: Me Intern in Penn Security Lab (SEAS) in 2001 while attending Drexel. Transitioned to full-time while earning a MSE in CIS. Moved to PSOM in 2006. Started building POP in late 2016.

Background: Organization PMACS (Penn Medicine Academic Computing Services) was created through the consolidation of various PSOM IT groups. Well over 100 IT professionals including system admins, infrastructure personnel, LSPs, application owners, etc. Have own our Active Directory domain serving Windows, Macs and Linux. Our helpdesk software is currently Quest K1000.

Background: Me + Organization Part of the Windows team. Primary responsibility is Active Directory and associated software infrastructure. I’m a strange hybrid of a programmer, system admin, project leader and evangelist. Only developer of POP.

Background: POP Main drivers for developing POP include… We receive many simple and repetitive requests. We had minimal integration both internally with our own apps and to University systems. Wanted a self-service portal for technical users to do work on operational systems in a controlled way. Requests: reset user account, add users to group, disable account, create account.

Background: POP, cont. Leading to a desire to… Automate as many simple and common tasks as possible. Make tasks requiring approval to be “single click”. Create new functionality in the enterprise by integrating our systems, Penn systems and Penn Medicine systems. Have auditable, repeatable and defined processes for as many tasks as possible.

Background: POP, cont. My solution to the problem statement laid out is… POP, the PMACS Operations Portal Let’s look at the design of POP at a high level…

POP: Design In its simplest form, POP is… A web-based application that presents forms to users so they may query or change our operational systems. Depending on security, users can self-service the request or it can be held for review. Supports many behind-the-scenes integrations to perform the operations in real time.

POP: Design, cont. Primary users of POP are… LSPs. System Admins. Application Owners. Managers.

POP: Design/Forms The basic design principle of POP is a form. A form is created for a discrete task. It has data fields. It has code that is executed on various events. It has ACLs.

POP: Design/Forms, cont. Forms can… Be held for approval by other people. Pull in functionality from other forms. Link with tickets in our helpdesk. Be activated by a variety of input methods. Inputs: web, scheduler, sms, email, slack

POP: Design/Forms, cont. Forms are entry points into POP. Users complete forms and submit them to a POP controller. Programming for the forms translate the form data and the requested action into actual operations on our systems. POP uses whatever layers are necessary to complete the requests. Layers: AD/LDAP, Oracle, RESTful services, web scraping

POP: Design/AD integration POP and AD are closely intertwined… POP is the primary AD manipulation tool in PMACS. Many AD account ops were a primary use case for early POP releases. As more apps were moved to AD, we needed tools to manage app users in AD. AD usernames were standardized to PennKeys.

POP: Design/AD integration, cont. POP uses AD for… Authenticating users. Verifying ACLs (based on group membership) on forms. Storing information needed to perform some automation.

POP: Design/Other Integrations There is much more to POP than just AD manipulation… Integrating with Penn Medicine for provisioning email accounts. Getting information from Penn Community. Sending SecureShare messages. And…

POP: Design/Other Integrations, cont. The operational groups of PMACS interact with a wide variety of systems on a daily basis. Most of these systems do not talk to each other. Penn Assignments

POP: Selected Workflows POP has over 300 forms. Exploring all of them in this forum is not feasible. I have selected a few workflows that I feel demonstrate the power and value of POP. Those workflows marked as Demo will be demonstrated live.

POP: Looking up a user Displaying user information is probably the most used form in POP. Provides basic user information from AD. Includes additional information from other operational systems such as reset and KACE. All combined in to one succinct view. Demo

POP: Creating a new user All AD user accounts are provisioned through POP. Requestor fills out a form with the usual information. Creates a helpdesk ticket for tracking purposes. Approvers see a split view: original form and Penn Community information. Approval of the form creates (and then executes) a plan. Demo

POP: Creating a new user, cont. New user plan includes… Creating the user account. Sending the user’s new password via SecureShare. Creating and setting ACLs for the user’s DFS home folder.

POP: Creating a new user, cont. New user plan also includes… Adding the user to any requested groups. Synchronizing the user’s AD account with Penn Community affiliation data. Key point: all this stuff happens the same way, regardless of who does the work. There is only one way to make a user in POP and, by proxy, in PMACS.

POP: Disabling users A user can be disabled in two ways… Interactive request by an authorized user. Triggered by changes in Penn Community. The latter is more interesting… so we’ll explore that.

POP: Disabling users, cont. POP monitors Penn Community for changes to user affiliations. Upon losing certain affiliations… POP disables the account. Removes all groups from the account, storing the removed groups in AD. Changes the login shell in /bin/false.

POP: Disabling users, cont. Account disables: fail secure vs fail open. Necessary given the number of users and decentralized nature. Easily reversed. Notifications are sent out.

POP: Local admin passwords PMACS uses LAPS (local administrator password solution) to further enhance Windows workstation security. Each machine in the domain has a different admin password. Password is stored in AD. LSPs can access password via POP.

POP: Local admin passwords, cont. Using POP “in the field” can be inconvenient… POP added an integration with a SMS provider, Plivo. LSPs can text POP from “known” cell phone numbers. POP will respond back to authorized users with the password.

POP: Local admin passwords, cont. Sample of SMS capabilities…

POP: Assignments Assignments is critical to both LSPs and sysadmins. Using ISC’s Assignments API, POP has webified most of the core functionality of Assignments. This is a generic use case – the integrated use case for PSOM use will not be discussed. We’re all pretty familiar with Assignments, so not much more will be said here. Demo

POP: Penn Medicine Email Integration PSOM leverages Penn Medicine’s O365 instance for email. LSPs request new accounts by creating tickets in Penn Medicine’s ticketing system. This process requires duplication of information and work. Working on a new integration with the Penn Medicine admin team.

POP: Penn Medicine Email Integration, cont. Old versus new… The way this looks like on the new usernew form… It is quite literally one check box for a LSP. Reiterates a key point of POP: simplicity.

POP: Penn Medicine Email Integration, cont. POP also updates the ISC online directory. POP scans for changes between Penn Medicine O365 accounts and Penn Community for PSOM people. Changes for PSOM people are pushed to ISC’s directory using a REST service.

POP: Demo time Let’s see POP in action… Looking up a user Creating a new user (LSP view / Admin view) Assignments

POP: Implementation POP is written in Perl and runs primarily under IIS. Secondary site runs under Apache on CentOS. Codebase is over 43k lines of code/configuration. 1 dev: me.

POP: Conclusion POP saves a lot of people in PSOM time and effort in their day to day jobs. Things that need to happen with certain operations are automatically done. Processes are predictable and repeatable. Logging is centralized. “There is usually a POP form for that…”

POP: Conclusion, cont. The way in which POP allows admins to approve requests… Allows admins to be more responsive to requests. Eliminates admin fat fingering. Enables admins to stay logged into one system instead of many.

POP: Conclusion, cont. Organizational gains… Details of operations become more formalized because they are backed by code. IT operations move towards standardization because they run on one system. One stop shopping makes life easier.

POP: Conclusion, cont. This is only the tip of the iceberg. I intentionally glossed over most of the technical details regarding the implementation… particularly security. There are lots of interesting things that you can do. I included only the most approachable in this talk.

POP: Question/Answer Session Thanks for your attention! Questions? Follow-up communication channels: luxk@upenn.edu @luxk on Slack https://kevinlux.info/ Contains older presentations to SUG and Security-SIG.