Class 10 CS 480-008 1 March 2016 On the board ------------ 1. Last time 2. OKWS, continued 3. Analysis and discussion 4. setuid 5. Survey --------------------------------------------------------------------------- 1. Last time --Unix's mechanisms for isolation and sharing processes permissions --OKWS 2. OKWS, continued How does OKWS enforce isolation between components in Figure 1? * okld runs each service with a separate UID and GID. So services can't read/write each other's memory. * okld uses chroot to confine each process to a separate directory (almost). Services can't read/write *any* files (system files, app state, etc.). pubd and oklog can only get at their own files. * Why is okld a separate process? Must run as superuser to bind to port 80, call chroot() and setuid(). We want as little code as possible to run as superuser. * Why is okd a separate process? --We need a way to route HTTP requests to the right svc. --okd sees all requests, so we don't want to do anything else in okd. --note: okd does *not* run as superuser; it is given port 80 by okld (in the form of an already bound socket) * Why is oklogd a separate process? --We don't want corrupt svc to delete/overwrite log files. --More generally we don't want svcs to have any access to files (too bug-prone). --So all file accesses go through a separate process [explain result of attacked svc: overwrite (ruled out) versus append-with-noise (possible)] * Why is pubd a separate process? --Keeps file handling code out of svcs. * Why are database proxies separate? Why not let svcs talk to the DB? Force communication through narrow interface: RPC [idea: "RPC communication channel is less error-prone that HTTP messaging."] Ensure that each service cannot fetch wrong data, if it is compromised. DB proxy protocol defined by app developer, depending on what app requires. Proxy enforces overall query structure (select, update), but allows client to fill in query parameters. Where does the 20-byte token come from? Passed as arguments to service. Who checks the token? DB proxy has list of tokens (& allowed queries?) Who generates token? Not clear; manual by system administrator? What if token disclosed? e.g. compromised chat svc could issue queries as e-mail service. and read/write any user's e-mail. * Table 1: why are all services and okld in the same chroot? Is it a problem? How would we decide? What are the readable, writable files there? Readable: shared libraries containing service code. Writable: each service can write to its own /cores/. Where's the config file? /etc/okws_config, kept in memory by okld (read on startup) oklogd and pubd have separate chroots because they have important state: oklogd's chroot contains the log file, want to ensure it's not modified. pubd's chroot contains the templates, want to avoid disclosing them (?). * Why a separate GID for each service? Need to execute binary, but file ownership allows chmod (and need to prevent services from doing chmod) Solution: file owner: root group owner: svc mode: 0410, meaning: svc can execute the binary but not read/write/chmod it root can read the binary but not write or execute it Why is root denied execute? (b/c of least privilege) this guards against bugs in okld, for instance the case that it forgot to drop its privilege after fork. if it did have that bug, the svc would now be running as root! Why can't service read its own binary? (least privilege again) Consider an attacker who compromises a service. If the binary is readable, then the attacker can very easily extract a credential from the binary, by exec()ing a shell and dumping the binary. If the binary is not readable, then the attacker has to do a little bit more work: it has to work within the address space of the existing process, and iterate over the heap, looking for tokens. 3. Analysis and discussion What harm comes if each component compromised? What's the "attack surface" of each? * okld: superuser access to web server machine, but maybe not directly to DB. attack surface: small (no user input other than svc exit). * okd: intercept/modify all user HTTP reqs/responses, steal passwords. attack surface: parsing the first line of HTTP request; control requests (repub, relaunch) * pubd: corrupt templates, leverage to maybe exploit bug in some service? attack surface: requests to fetch templates from okd. * oklogd: corrupt/ignore/remove/falsify log entries. attack surface: log messages from okd, okld, svcs * service: send garbage to user, read/write service's data _for any user_, send requests to dbproxy. attack surface: HTTP requests from users (+ control msgs from okd) * dbproxy: access/change all user data in the database it's talking to. attack surface: requests from authorized services requests from unauthorized services (easy to drop) NOTE: OS kernel is part of the attack surface once a single service is compromised. Linux kernel vulnerabilities rare, but still show up several times a year. OKWS assumes developer does the right thing at design level (maybe not impl): --Split web application into separate services (not clump all into one). --Define precise protocols for DB proxy (otherwise any service gets any data). Performance? --Seems better than most alternatives. --Better performance under load (so, resists DoS attacks to some extent) How does OKWS compare to Apache? --Overall, better design. --okld runs as root, vs. nothing in Apache, but probably minor. --Neither has a great solution to client-side vulnerabilities (XSS, ..) Where should an attacker look for weaknesses? Probably lots of bugs in svc implementations: buffer overflows, logic bugs --Maybe not so bad for the "friend status" service --Bad that bug in e-mail svc means I can read/write your e-mail --Very bad if bugs in the password authentication service --Hopefully sensitive services have few lines of code (== few bugs) Bugs in OS kernel --Code injected into a svc might be able to exploit a kernel bug to become superuser, break out of chroot. Bugs in okd URL parsing Bugs in DB proxies (SQL injection, too permissive). --OKWS good example of privilege separation on Unix, and the hoops that one has to jump through. --Why not process per user? Or even process per user per service? Per-service isolation probably made sense for okcupid given their apps. (i.e., perhaps they need a lot of sharing between users anyway?) Per-user isolation requires allocating UIDs per user, complicating okld, and reducing performance (though may still be OK for some use cases). **Issue of memory and overhead consumed by fork()ing processes (bigger issue in 2003) --How have these ideas evolved since 2004, when OKWS was published? The problems are pretty common, still --OKWS itself still used at okcupid.com, but not adopted elsewhere (though I used it for a grad project, and it was fine). Why? --C++ not a very popular language for web programming (don't need its performance...the zook web server uses Python...). --Too much app-specific effort to partition services finely. --Too much app-specific effort to build DB proxy interfaces (and DB proxy is a deep part of the OKWS story) [Can be hard to precisely define the allowed DB queries ahead of time. (Although if it's hard, might be a flag that security policy is fuzzy.)] --What do today's systems do? A: Partition at coarse grain. (Credit card processing companies split credit card data vs. everything else.) Use virtual machines or physical machine isolation to split apps, DBs, .. client/server, browser/server, VMs, Linux containers. Often partitioning is by independent service / application / web site. --Some systems use fine-grained cooperating partitions, reminiscent of OKWS. ssh-agent process to hold crypto keys, vs ssh itself. Chrome runs each frame in an isolated process. It takes a lot of work to build in this style. --How could you integrate modern Web application frameworks with OKWS? Need to help okd figure out how to route requests to services. Need to implement DB proxies, or some variant thereof, to protect data. Depends on how amenable the app code is to static analysis. Or need to ask programmer to annotate services w/ queries they can run. Need to ensure app code can run in separate processes (probably OK). STUDENT QUESTIONS --can you clarify an aspect of oklogd? if "oklogd runs as an unprivileged user in its own chroot environment", then it "cannot maliciously overwrite or truncate log files; it would only have the ability to fill them with 'noise'". How can it fill the logs with noise if it does not have write permissions? Even if it could fill it with "noise", how would okd distinguish between noise an actual logs? [via messages to the log daemon. okd wouldn't do the distinguishing; a human would have to.] --I was a bit confused about the section regarding okld catching when processes exit abnormally, and will change the uid and gid of the core files, before restarting the process. It would be helpful to clarify on how this process is carried out. [okld uses its root privileges to mod the perms (via something like chmod) of the core files.] --Additional clarification question.  Under section 5.4, the author states that OKWS has no reason to support keep-alive connections as all pages generated contain static content. [it's that all pages generated contain *dynamic* content] But the author also says that the servers hosting the static content can enable HTTP keep-alive.  Why is that?  Does the page not send the static content then close the connection since all the content is static? [keep-alive is more important for static content because the overhead to create a new connection matters more there.] --Question: The paper mentions that forking a new process for each user would be the most secure option, but is not scalable. Is this indeed the ideal solution, and is it more feasible nowadays with the advances we have made since the early 2000s when the paper was written? If not, in the future? [good question. see the discussion above. it is more feasible, and it is probably the way you might build things today.] --Question: I didn’t quite understand the section explaining “process isolation” and the comparisons of Okws to other servers like Apache, Flash, Haboob, and Strict, especially with regards to the number of concurrent HTTP connections and process pool sizes.  --Would a RELAUNCH RPC issued by a site maintainer causes the relaunch of all child services? Are there any security problems with the "administration client"? --Why race condition could potentially contribute to privilege escalation? --What specifically makes OKWS slow at serving static content, especially when compared to Flash? [it's just not optimized for that: no cache, etc.] Also, the authors write that they rely on other server code such as Apache or Flash to serve static content. In this case, what is the exact purpose of pubd? [pubd serves *templates*, which are a component of dynamic content.] Is the "parsing and caching" that pubd does on startup reading from a static content server and caching those files OKWS will need? [this activity refers to reading from the file system] --The paper mentions that PHP scripts that allow sequences such as ".." could allow access to files higher up in the file system. In an insecure environment, the PHP program that runs these scripts a child of some process that does have read permissions? Could you give the PHP program execute permissions without read permissions so that even if this sequence was allowed, the remote client would not have read access? --Why is isolation like this not common practice already? --Realistically, is OKWS's performance 'good' considering it is being compared against general purpose Web servers? --Do you know the dating site that the OKWS was tested for? --What is the benefit of separating static and dynamic content? --How influential is the use of RPC protocol in the OKWS? It is listed as a Security Benefit, but what makes RPC so beneficial over other options? --------------------------------------------------------------------------- Further reading on privilege separation: "Make Least Privilege a Right Not a Privilege", Krohn et al., HotOS 2005 [picks up where OKWS left off. uses the hoops that OKWS has to jump through to motivate new OSes; see research OSes below] https://www.usenix.org/legacy/event/hotos05/final_papers/full_papers/krohn/krohn.pdf "Preventing Privilege Escalation", Provos, Friedl, Honeyman on Privilege-Separated OpenSSH, Proc. Usenix Security, 2003 [This is a classic; it's how openssh is implemented. It also explains the mechanism of file descriptor passing, and relates it to capabilities. This paper is related to OKWS.] http://www.peter.honeyman.org/u/provos/papers/privsep.pdf "Privman: A Library for Partitioning Applications", Kilpatrick, Usenix Technical, 2003 [The name says it all.] https://www.usenix.org/legacy/events/usenix03/tech/freenix03/full_papers/kilpatrick/kilpatrick.pdf "Wedge: Splitting Applications into Reduced-Privilege Compartments", A. Bittau, P. Marchenko, M. Handley, and B. Karp., Proc. NSDI, 2008 [A great read that works through the mechanics of fine-grained partitioning] https://www.usenix.org/legacy/event/nsdi08/tech/full_papers/bittau/bittau.pdf Research OSes that incorporate DIFC (Distributed Information Flow Control): Asbestos "Labels and Events in the Asbestos Operating System", Efstathopoulos et al., SOSP 2005 HiStar "Making Information Flow Explicit in HiStar", Zeldovich et al., OSDI 2006 Flume "Information Flow Control for Standard OS Abstractions", Krohn et al., SOSP 2007 One of the key motivating examples for these research OSes was OKWS, namely that in order to use Unix is a least-privileged way, one has to jump through hoops. That motivated the search for better, simpler primitives. This in turn led to a research craze on DIFC (lasting from 2005-2009), which revisited ideas from the 1970s surrounding building hardened operating systems for defense applications. --------------------------------------------------------------------------- 4. setuid --Some legitimate actions require more privs than UID --E.g., how should users change their passwords? --Passwords are stored in root-owned /etc/passwd and /etc/shadow files (see above) --setuid/setgid are the sole means on Unix to *raise* a process's privilege level --Solution: Setuid/setgid programs --special "setuid" bit in the permissions of a file --Run with privileges of file's owner or group --Each process has _real_ and _effective_ UID/GID (you have seen effective UIDs show up in lab3a, in the setresuid() syscall) -- _real_ is user who launched setuid program -- _effective_ is owner/group of file, used in access checks --for setuid programs, on exec() of binary, kernel sets effective uid = file uid --examples: [mig ~]$ ls -l `which ping` -rwsr-xr-x 1 root root 35680 2010-03-12 05:41 /bin/ping* [my-local-computer ~]$ ls -l `which passwd` -r-sr-xr-x 1 root wheel 70352 Jun 18 2009 /usr/bin/passwd* [note the 's'] --Obviously need to own file to set the setuid bit --Need to own file and be in group to set setgid bit ['s' is in place of 'x' because it's not meaningful to talk about the setuid bit unless the file was executable in the first place [since setuid governs what happens when a process exec()s the file]] --(idea: a way for root -- or another user -- to delegate its ability to do something.) --Examples: --/usr/bin/passwd : change a user's passwd. User needs to be able to run this, but only root can modify the password file. --/bin/su: change to new user ID if correct password is typed. --/sbin/ping: users need to be able to ping but only root is allowed to open "raw sockets" (example of incoherent security model; but very hard to fix) --Have to be EXTREMELY careful when writing setuid code [Imagine that you are writing or installing a setuid program. In other words, you are willing to let that program run with *your* privileges.] --Fundamental reason you need to be careful: very difficult to anticipate exactly how and in what environment the code will be run....yet when it runs, it runs with *your* privileges (where "your" equals "root" or "whoever set the setuid bit on some code they wrote") --NOTE: Attackers can run setuid programs any time (no need to wait for root to run a vulnerable job) --FURTHER NOTE: Attacker controls many aspects of program's environment --Example attacks: --Close fd 2 before execing program --now, setuid program opens a file, for example the password file.... (normally, would be fd=3, but because fd 2 was closed, the file will be given fd 2). --then, the program later encounters an error message and does fprintf(stderr, "some error msg"). --result: the error message goes into the password file! --fix: for setuid programs, kernel will open dummy fds for 0,1,2 if not already open --Set maximum file size to zero (if, say, setuid program changes a password and then rebuilds some password database) --IFS hackery --say there's a program called "preserve" installed as setuid root; this setup was used by old editors to make a backup of files in a root-accessible directory --preserve runs system("/bin/mail"). [to mail error messages or whatever] --"system" uses the shell to parse its argument --now if IFS (internal field separator) is set to "/" before running vi, then we get the following: --vi forks and execs /usr/lib/preserve (IFS is still set to '/', but exec() call doesn't care) --preserve invokes system("/bin/mail"), but this causes shell to parse the arguments as: bin mail --which means that if the attacker locally had a malicious binary called 'bin', then that binary could do: cd /homes/mydir/bin cp /bin/sh ./sh chown root sh # this succeeds because 'bin' is running as root chmod 4755 sh # this succeeds because 'bin' is running as root --result is that there is now a copy of the shell that is owned by root and setuid root --anyone who runs this shell has a root shell on the machine --fix: shell has to ignore IFS if the shell is running as root or if EUID != UID. --also, modern shells refuse to run scripts that are setuid. (the issue there is a bit different, but it is related.) --ptrace() examples [we covered these in cs202...placing them in the notes for reference...please ask the course staff if you have questions.] What is ptrace()? It's a system call, used by debuggers Lets one process modify another's memory Attack 1: --attacker ptraces setuid program P --P runs with root's privileges --now manipulate P's memory, get arbitrary privilege on the machine. this is bad. --fix: don't let process ptrace more privileged process or another user's process for example, require sender to match real and effective UID of target Attack 2: --attacker owns two unprivileged processes A and B. --A ptraces B. so far, so good. no violation of the rule above. --Then B execs a setuid program (for example, "su whatever"), which causes B's privilege to be raised. (recall that the "su" program is setuid root. "su pat" becomes user "pat" if someone types pat's password.) --Now A is connected to a process that is running with root's privileges. A can use B's elevated privileges. This is bad. --fix: disable/ignore setuid bit on binary if ptraced target calls exec() --> but let root ptrace anyone Attack 3: --now, say that A and B are unprivileged processes owned by attacker --say A ptraces B. so far, so good. no violation of prior two rules. --say A executes "su attacker", i.e., it's su'ing to its own identity --While su is superuser, B execs "su root" --remember, the attacker programmed B, and can arrange for it to exec the command just above. --BUT! remembering the ptrace rules above, the ptrace succeeds. the reason is that at this moment A is the superuser, so no problem with B's exec() honoring the setuid. --attacker types password into A, gets shell, and now this (unprivileged) shell is attached to "su root". --the attacker can now manipulate B's memory (disable password checks, etc.) so that the "su root" succeeds, at which point A is connected to a root shell See Linux Yama module as a partial defense: https://www.kernel.org/doc/Documentation/security/Yama.txt Another issue: --consider a setuid process that does a bunch of privileged things and then drops privileges to become user again --should be okay, right? *****--NO. once the process has seen something privileged and then become the user again, it can be ptraced(), and the confidential things it has seen (or the privileged resources that it holds) can be manipulated by an unprivileged user.**** --fix? privilege separation Another class of attacks: TOCTTOU (time-of-check-to-time-of-use) --very common attack --say there's a setuid program that needs to log events to a file: fd = open(logfile, O_CREAT|O_WRONLY|O_TRUNC, 0666); --what's the problem? --setuid program shouldn't be able to write to file that user can't. thus: if (access(logfile, W_OK) < 0) return ERROR; fd = open(logfile, ....) should fix it, right? NO! --here's the attack........ setuid program attacker creat("/tmp/X"); check access("/tmp/X") --> OK unlink("/tmp/X"); symlink("/tmp/X", "/etc/passwd") open("/tmp/X") --from the BSD man pages: "access() is a potential security hole and should never be used." --the issue is that access check and open are non-atomic --to fix this, have to jump through hoops: manually traverse paths. check at each point that the dir you're in is the one you expected to be in (i.e., that you didn't accidentally follow a symbolic link). maybe check that path hasn't been modified also need to use APIs that are relative to an opened directory fd: -- openat, renameat, unlinkat, symlinkat, faccessat -- fchown, fchownat, fchmod, fchmodat, fstat, fstatat Or Wrap groups of operations in OS transactions --Microsoft supports transactions on Windows Vista and newer https://msdn.microsoft.com/en-us/library/windows/desktop/bb986748%28v=vs.85%29.aspx --research papers: http://www.fsl.cs.sunysb.edu/docs/valor/valor_fast2009.pdf http://www.sigops.org/sosp/sosp09/papers/porter-sosp09.pdf --------------------------------------------------------------------------- setuid refs: http://css.csail.mit.edu/6.858/2014/readings/setuid.pdf http://httpd.apache.org/docs/trunk/suexec.html --------------------------------------------------------------------------- Acknowledgment: MIT's 6.858 staff