1. Nmap Scripting Engine Patrick Donnelly University of Notre Dame Nmap Project Developer

2. Nmap Introduction ● Nmap is a Network Scanning/Mapper for security auditing and network exploration. ● Used to identify services on a host, the operating system running on the host, network topology. ● Nmap offers a variety of types of scans in order to scan a host for its open services. (-sP, -sS, -sA, -sV, -sC, -O)

3. Nmap Example batrick@batbytes:~$ nmap -v scanme.nmap.org Starting Nmap 5.05BETA1 ( http://nmap.org ) at 2009-10-06 07:58 EDT Initiating Ping Scan at 07:58 Scanning 64.13.134.52 [2 ports] Completed Ping Scan at 07:58, 1.42s elapsed (1 total hosts) Initiating Parallel DNS resolution of 1 host. at 07:58 Completed Parallel DNS resolution of 1 host. at 07:58, 0.08s elapsed Initiating Connect Scan at 07:58 Scanning scanme.nmap.org (64.13.134.52) [1000 ports] Discovered open port 80/tcp on 64.13.134.52 Discovered open port 53/tcp on 64.13.134.52 Completed Connect Scan at 07:58, 4.25s elapsed (1000 total ports) Host scanme.nmap.org (64.13.134.52) is up (0.042s latency). Interesting ports on scanme.nmap.org (64.13.134.52): Not shown: 994 filtered ports PORT STATE SERVICE 25/tcp closed smtp 53/tcp open domain 70/tcp closed gopher 80/tcp open http 113/tcp closed auth 31337/tcp closed Elite Read data files from: /usr/local/share/nmap Nmap done: 1 IP address (1 host up) scanned in 6.00 seconds

4. Another Nmap Example batbytes:~# nmap -p- -sV -O localhost Starting Nmap 5.05BETA1 ( http://nmap.org ) at 2009-10-06 08:10 EDT Interesting ports on localhost (127.0.0.1): Not shown: 65529 closed ports PORT STATE SERVICE VERSION 22/tcp open ssh OpenSSH 5.1p1 Debian 5 (protocol 2.0) 80/tcp open http Apache httpd 2.2.9 ((Debian)) 443/tcp open ssh OpenSSH 5.1p1 Debian 5 (protocol 2.0) 3690/tcp open svnserve Subversion 8080/tcp open http-proxy Squid webproxy 2.7.STABLE3 22222/tcp open ssh OpenSSH 5.1p1 Debian 3ubuntu1 (protocol 2.0) Device type: general purpose Running: Linux 2.6.X OS details: Linux 2.6.15 - 2.6.27 Network Distance: 0 hops Service Info: OS: Linux OS and Service detection performed. Please report any incorrect results at http://nmap.org/submit/. Nmap done: 1 IP address (1 host up) scanned in 10.19 seconds

5. A Brief History of Nmap ● September 1, 1997 Nmap is first released in Phrack Magazine, 2000 lines long. ● December 12, 1998, Nmap 2.0 is released with OS detection. ● September 16, 2003 Nmap with service detection is released in 3.45. ● May 24, 2006 Diman Todorov is sponsored in the Google Summer of Code to create the Nmap Scripting Engine. ● December 10, 2006 NSE is publicly released in Nmap 4.21ALPHA1.

6. Nmap Scripting Engine ● Needed to handle specific tasks that cannot be done easily in Nmap in C++. ● Motivations for using Lua. ● NSE uses Lua to allow scripts to run against a host in parallel. ● Scripts use bindings to Nmap libraries for performance and access to networking facilities.

7. Nmap Scripting Engine Goals/Tasks ● Advanced Network discovery – WHOIS, identd lookups. ● Sophisticated Version detection – Skype 2 requires two probes. ● Vulnerability detection – smb vulnerabilites. ● Backdoor detection (Worms) – p2p-conflicker.nse ● Vulnerability exploitation, possible but not actively. ● Service exploration – html spider.

8. NSE in Action batbytes:~# nmap -p- -sV -O --script all localhost Starting Nmap 5.05BETA1 ( http://nmap.org ) at 2009-10-06 08:29 EDT Interesting ports on localhost (127.0.0.1): Not shown: 65529 closed ports PORT STATE SERVICE VERSION 22/tcp open ssh OpenSSH 5.1p1 Debian 5 (protocol 2.0) |_ banner: SSH-2.0-OpenSSH_5.1p1 Debian-5 | ssh-hostkey: 1024 66:b6:48:76:64:9c:73:0e:20:65:28:8b:70:52:02:8a (DSA) |_ 2048 d7:e0:24:7f:75:8f:d9:64:47:b5:2b:ed:a5:66:4b:7f (RSA) 80/tcp open http Apache httpd 2.2.9 ((Debian)) |_ http-favicon: Invalid favicon: Empty File | http-headers: (HEAD used) | HTTP/1.1 200 OK | Date: Tue, 06 Oct 2009 12:29:13 GMT | Server: Apache/2.2.9 (Debian) | Last-Modified: Sun, 30 Aug 2009 06:07:12 GMT | ETag: "84a8-0-47255ba67e400" | Accept-Ranges: bytes | Content-Length: 0 | Vary: Accept-Encoding | Connection: close |_ Content-Type: text/html |_ http-date: Tue, 06 Oct 2009 12:29:13 GMT; 0s from local time.

9. NSE in Action (2) 443/tcp open ssh OpenSSH 5.1p1 Debian 5 (protocol 2.0) |_ banner: SSH-2.0-OpenSSH_5.1p1 Debian-5 | ssh-hostkey: 1024 66:b6:48:76:64:9c:73:0e:20:65:28:8b:70:52:02:8a (DSA) |_ 2048 d7:e0:24:7f:75:8f:d9:64:47:b5:2b:ed:a5:66:4b:7f (RSA) 3690/tcp open svnserve Subversion |_ banner: ( success ( 2 2 ( ) ( edit-pipeline svndiff1 absent-entries... 8080/tcp open http-proxy Squid webproxy 2.7.STABLE3 |_ http-date: Tue, 06 Oct 2009 12:29:13 GMT; 0s from local time. | http-open-proxy: Potentially OPEN proxy. |_ Methods supported: GET HEAD 22222/tcp open ssh OpenSSH 5.1p1 Debian 3ubuntu1 (protocol 2.0) |_ banner: SSH-2.0-OpenSSH_5.1p1 Debian-3ubuntu1 | ssh-hostkey: 1024 d5:44:d2:fa:15:7c:81:a7:d2:17:7f:9d:10:3f:b2:86 (DSA) |_ 2048 02:4b:e4:64:5f:76:88:11:21:e8:19:5e:ea:6b:72:85 (RSA) Device type: general purpose Running: Linux 2.6.X OS details: Linux 2.6.15 - 2.6.27 Network Distance: 0 hops Service Info: OS: Linux OS and Service detection performed. Please report any incorrect results at http://nmap.org/submit/. Nmap done: 1 IP address (1 host up) scanned in 26.38 seconds

10. Scripts asn-query.nse http-trace.nse smb-enum-users.nse auth-owners.nse http-userdir-enum.nse smb-os-discovery.nse auth-spoof.nse iax2-version.nse smb-pwdump.nse banner.nse imap-capabilities.nse smb-security-mode.nse daytime.nse irc-info.nse smb-server-stats.nse dhcp-discover.nse ms-sql-info.nse smb-system-info.nse dns-random-srcport.nse mysql-info.nse smtp-commands.nse dns-random-txid.nse nbstat.nse smtp-open-relay.nse dns-recursion.nse p2p-conficker.nse smtp-strangeport.nse dns-zone-transfer.nse pjl-ready-message.nse sniffer-detect.nse finger.nse pop3-brute.nse snmp-brute.nse ftp-anon.nse pop3-capabilities.nse snmp-sysdescr.nse ftp-bounce.nse pptp-version.nse socks-open-proxy.nse ftp-brute.nse realvnc-auth-bypass.nse sql-injection.nse html-title.nse robots.txt.nse ssh-hostkey.nse http-auth.nse rpcinfo.nse sshv1.nse http-date.nse script.db ssl-cert.nse http-enum.nse skypev2-version.nse sslv2.nse http-favicon.nse smb-brute.nse telnet-brute.nse http-headers.nse smb-check-vulns.nse upnp-info.nse http-iis-webdav-vuln.nse smb-enum-domains.nse whois.nse http-malware-host.nse smb-enum-processes.nse x11-access.nse http-open-proxy.nse smb-enum-sessions.nse http-passwd.nse smb-enum-shares.nse

11. NSE Script Makeup/Format ● String Fields – author, description, license ● runlevel – Ordered execution ● Categories – Table of categories script qualifies for (e.g. “vuln”, “discovery”, intrusive”) ● portrule or hostrule functions – service/port dependence. ● action function – heart of the script.

12. Simple Example author = "Patrick Donnelly " license = "Same as Nmap--See http://nmap.org/book/man-legal.html" categories = {"demo", "safe"} runlevel = 1.0 description = "Example script grabbing the 'banner' of a service" function portrule (host, port) return port.protocol == 'tcp' end function action (host, port) local socket = nmap.new_socket() socket:set_timeout(2500) -- milliseconds assert(socket:connect(host.ip, port.number)) local status, line = assert(socket:receive_lines(1)) return line end

13. NSE Implementation ● Parallelism, Parallelism, and Parallelism! ● Coroutines to facilitate parallelism. ● Each script may spawn multiple threads that may run against many different hosts “simultaneously”. ● Nsock library binding yields on blocking calls.

14. Essential Structure of the Main Loop ● The main loop for NSE continuously runs scripts from a “running” queue. ● A script yields when it connects on a socket, receives data on a socket, sends data, etc. Thread is placed in the waiting queue. ● Loop ends when both the running and waiting queue are empty. ● When the running queue is empty, we block on a select()-like call “nsock_loop”.

15. The Nsock binding ● The Nsock library is the performance critical, parallel socket library Nmap uses to perform massively parallel scans. ● NSE leverages the power of the Nsock library with a simple binding that offers a socket create function, nmap.new_socket() and many methods on these sockets. – socket:connect() – socket:receive() – socket:receive_lines() – socket:send()

16. Nsock Binding – Working in the Background ● Each blocking call will give a callback to Nsock in order to resume the thread when Nsock has finished the operation. ● The call will then yield control back to NSE. ● Nsock will periodically finish blocked calls by executing the registered callback. ● The thread is moved from the waiting queue to the running queue with the return values of the blocked call being passed to resume.

17. The connect function (method) ● Guided look at the code.

18. Problem with yielding threads ● The user script runs as though it is sequential and is completely unaware that it is running alongside other scripts. ● Script sees a blocking call but is actually a non- blocking call. ● Script may try to use coroutines for its own parallelism or use pcall which they can't yield across. ● The script must yield back to NSE, not to a master script coroutine!

19. Solutions for Script Coroutines ● The chain of coroutines the script may use for its own collaborative multithreading are yielded in a chain. ● A special, unique value is used to signal a yield initiated by NSE. coroutine.resume is hooked so NSE initiated yields are propagated back to NSE. The chain is resumed when NSE resumes the base (master) script coroutine. ● What about parallel script worker threads?

20. Scripts using pcall? ● Right now, functions that may yield (e.g. socket:connect()) will not throw an error. Instead a status is first returned. ● Resumable VM in 5.2?

21. NSE Main Functions ● Originally implemented entirely in C(++). ● Very inflexible and painful to maintain. ● The majority of NSE's internal data structures were in Lua inside the Lua Registry. The data took a lot of lines to create and utilize. ● Plagued with Segfaults – most developers weren't very familiar with Lua in C.

22. NSE Main in Lua ● Now use Lua to do all the work for initialization, dispatch, modifying state, parsing arguments. ● C side binds functions for Nmap facilities to use in the main NSE function. ● C side offers an API to access the Lua engine. ● (Guided tour of main loop code)

23. NSE abstractions for Scripts ● The Script is a class used to provide classic OOP facilities for the main NSE function. (A nice abstraction made possible by the switch to a Lua engine). ● Threads (another class) are created from scripts using Script:new_thread(). ● Scripts (class) are created and initialized at startup by the categories or files chosen on the command line. ● Threads (class) are created at runtime and run based on the return values of the hostrule or portrule.

24. Problems with Parallelism ● One usual concern when using any type of multithreading with resource contention is the possibility for deadlock. ● We have had a lot of problems with scripts deadlocking on resources. – # of sockets in the pool – Mutexes (discussed next) ● Solution: each Thread can get as many sockets as it wants (suboptimal).

25. Mutexes ● Scripts often have many threads running against multiple hosts, using the same resource. – WHOIS – Caches ● Mutexes are used to provide mutual exclusion so only one thread can be using a resource, other threads can use results or obtain a lock to use the resource. ● Problem: Deadlock! Solution: Resource cleanup.

26. Threads needing more Parallelism ● Right now we only have portrules and hostrules to facilitate parallel instances of scripts. ● A web server spider script (a work in progress) would desire having multiple worker threads working in parallel. ● Coordination? Condition Variables! – Similarities to preemptive multithreading problems

27. Essential Ideas ● Collaborative multithreading can experience the same problems with resource contention as traditional preemptive multithreading. ● We also solve these problems using mutual exclusion and condition variables.

28. Questions?