Presentation is loading. Please wait.

Presentation is loading. Please wait.

Cluster Computing at IQSS Alex Storer, Research Technology Consultant.

Similar presentations


Presentation on theme: "Cluster Computing at IQSS Alex Storer, Research Technology Consultant."— Presentation transcript:

1 Cluster Computing at IQSS Alex Storer, Research Technology Consultant

2 What is the RCE?  Research Computing Environment  For research in the social sciences  Get an account!

3 How do I access it?

4

5

6

7 What are your needs? Gigantic Process Many Processes

8 Gigantic Process RCE Powered

9 Gigantic Process

10 RCE Powered

11 Many Processes InputOutput

12 Many Processes Input 1Output 1 Input 2Output 2 Input 3Output 3 Input 4Output 4 Input 5Output 5

13 Many Processes

14 Example: Simulating  You have a model that takes 30 minutes to run and computes a result  You want to establish confidence intervals for this number by running it many times

15 Example.submit file

16 What command do I run?

17 Example.submit file What arguments do I give to the command?

18 Example.submit file What input do I give to the command?

19 Example.submit file Where do I save the outputs?

20 Example.submit file How many times do I run this?

21 Example.submit file /usr/bin/R --no-save --vanilla out.1 /usr/bin/R --no-save --vanilla out.2.. /usr/bin/R --no-save --vanilla out.10 Unix Standard Input/Output

22 How to submit a file condor_submit simulate.submit

23 Your out.0 file It's just everything that R writes to the screen from the script!

24 Example: Simulating  Instead of using the out.$(Process) structure, you can save the data in your script  You cannot expect the Processes to complete in order!  You shouldn't write to the same file until all processes are complete  Instead of calling a script, use a function

25 Example: Simulating (with a function) procid is an input We tell the function what to save and where to save it

26 Example: submitting a function in batch Execute this command in R. Specifically, run the simfunction.R file which we defined on the previous slide.

27 Example: submitting a function in batch Execute this command in R. Specifically, call the function sim.function with the input as $(Process). sim.function(0) sim.function(1) …

28 Example: submitting a function in batch We are no longer using the standard input and standard output, so we can leave these blank.

29 MATLAB Example  An example where we need to do the same thing to a number of data files and write out the results  Call a function which knows how to map the process ID to the data to load

30 MATLAB function

31 We will pass $(process) as the function input

32 MATLAB function Try to load: data_0.ma t data_1.ma t etc.

33 MATLAB function Compute the relevant result.

34 MATLAB function Save the results as: result_0.mat result_1.mat etc.

35 .submit File Example  The Arguments section is the most important, let's look at each piece individually

36 .submit File Example Arguments = "-nodisplay –singleCompThread –r ''" Start the arguments with double quotes (")

37 .submit File Example Arguments = "-nodisplay –singleCompThread –r ''" -nodisplay tells Matlab to not pop up the GUI

38 .submit File Example Arguments = "-nodisplay –singleCompThread –r ''" -singleCompThread tells Matlab to use only one core (this is what condor expects)

39 .submit File Example Arguments = "-nodisplay –singleCompThread –r ''" -r tells Matlab to execute whatever commands come next.

40 .submit File Example Arguments = "-nodisplay –singleCompThread –r ''" Put the commands to run in single quotes.

41 .submit File Example  Do NOT try to write your entire Matlab script in the submit file!  Some arguments must be executed before calling your script, however…

42 .submit File Example Arguments = "-nodisplay –singleCompThread –r ''" The commands to Matlab will go on inside the single quotes They must be on a single line!

43 .submit File Example Arguments = "-nodisplay –singleCompThread –r ''" setenv(''HOME'',''nfs/home/A/astorer''); cd(''/nfs/home/A/astorer/Work/outreach/matlab''); mytest($(PROCESS))

44 .submit File Example Arguments = "-nodisplay –singleCompThread –r ''" setenv(''HOME'',''nfs/home/A/astorer''); cd(''/nfs/home/A/astorer/Work/outreach/matlab''); mytest($(PROCESS)) setenv is required for Matlab to load your local preferences. You must use two single quotes instead of one single quote. Remember to set your own home directory, e.g. nfs/home/J/jdoe

45 .submit File Example Arguments = "-nodisplay –singleCompThread –r ''" setenv(''HOME'',''nfs/home/A/astorer''); cd(''/nfs/home/A/astorer/Work/outreach/matlab''); mytest($(PROCESS)) Change to the directory that contains the script you want to run.

46 .submit File Example Arguments = "-nodisplay –singleCompThread –r ''" setenv(''HOME'',''nfs/home/A/astorer''); cd(''/nfs/home/A/astorer/Work/outreach/matlab''); mytest($(PROCESS)) Finally, run the function on the $(PROCESS) variable.

47 Example submission  Because our function reads in data, we generate the data ahead of time  This is what is in our directory before submitting  ( ls lists directory contents)  Notice that we count starting from 0 !

48 Example submission  Use condor_submit to submit the file  Depending on the job, this may take some time to complete!

49 Example submission  Use condor_q to check the status of your jobs  Use condor_rm to clear your jobs.

50 Example submission  Use condor_q to check the status of your jobs  When this returns with no result, your jobs are complete.

51 Example submission  Results!

52 Stata Example Universe = vanilla Executable = /usr/local/bin/stata-mp Arguments = do notification = Complete notify_user = input = Test.do output = Test.out error = Test.err Log = Test.log Queue 1 This is like running stata-mp do Test.do

53 Notification! Universe = vanilla Executable = /usr/local/bin/stata-mp Arguments = do notification = Complete notify_user = input = Test.do output = Test.out error = Test.err Log = Test.log Queue 1 You can get s when your job is done!

54 Debugging  If your results aren't as expected, first check the error files

55 My jobs never finish?!  Sometimes, jobs aren't well formed and condor won't know what to do  Condor will hold these jobs  Your submit file is probably wrong somehow – try looking at the log file as well as the submit file H stands for "Held"


Download ppt "Cluster Computing at IQSS Alex Storer, Research Technology Consultant."

Similar presentations


Ads by Google