1. Matlab DIY Lesson 3: Parsing data

2. Today's Lesson “ parse ” = analyze into its parts o Sort the data you want MATLAB to use o Designate different segments of the data o Tell MATLAB to do different things with different parts of the data “ batch ” = process a group of records as a single unit (one command processes multiple files)

3. Why Parse? Let's look at a few example files: - data output may not be in evenly spaced rows - you might be interested in a subset of the data and want to separate it from the rest - you might want to run different analysis on different parts of the data

4. Your first.m program During previous lessons you typed individual commands and entered them in the command window.m allows you to access a series of commands by entering the name of the program in the command window Open MATLAB  set current directory  open current directory window  double click on L3_csvread_pc.m

5. L3_csvread_pc.m review commands from previous lessons

6. Parse option 1: choose rows and columns of data where Matlab begins to read data Reading in the data file data = csvread([inpath,infilename],1,1); tsvread xlsread dlmread These all read in numbers NOT text 0,0 = first cell in upper left corner 1,0 = can be used to eliminate header data 10,5 = starts reading on 10 th line, 5 th column

7. Common csvread errors >> d = csvread([inpath,filename],1,0); ??? Error using ==> csvread Filename must be a string. Step 1: see what Matlab thinks filename is: 1) type filename in command window 2) check class of filename in workspace Step 2: adjust so that Matlab sees filename as a string 1) redefine filename variable by adding ‘’ or [] 2) type filename{1} this turns it into a string d=csvread([inpath,filename{1}],1,0);

8. Common csvread errors >> u = csvread([inpath,filename{1}],10,0); ??? Error using ==> csvread File not found. Step 1: check the path -is the name what you think it is? -have you been consistent in spelling for inpath and filename? Step 2: check your data file -have you used the right name? -should it be in a folder? -have you used the right suffix? u = csvread([data_inpath,filename{1}],10,0);

9. Common csvread errors >> c = csvread([data_inpath,filename{1}],0,0); ??? Error using ==> textscan Mismatch between file and format string. Trouble reading number from file (row 1, field 1) ==> time, Step 1: open data file and check for letters in the data Step 2: adjust to avoid problem area --adjust where Matlab begins reading or --adjust organization of the file

10. Parse option #2 Create a new smaller array with a subset of data >> c = csvread([data_inpath,filename{1}],1,1); >> k = csvread([data_inpath,filename{1}],2:8,1:10); ??? Error using ==> textscan Header lines must be a scalar integer. Error in ==> csvread at 45 m=dlmread(filename, ',', r, c); >> k = c(2:8,1:10);

11. Parse option #3: Naming segments of the data x_pos=data(:,1); y_pos=data(:,2); z_pos=data(:,3); mus1=data(:,4); mus2=data(:,5); mus3=data(:,6); event=data(:,7); time=data(:,8); %notice that we started reading the data at 1,1 so the file data is missing the first row and first column, each of these column numbers have been adjusted accordingly

12. Now you can perform simple formulas mn_x=mean(x_pos); SD_y=std(y_pos); Two ways to find more info on abbreviations for mathematical functions 1) command window >>help mean 2) Go to Matlab help and search for mean

13. Parse option #4: create program to select a subset of data onstamp=find(event>4); ontime=onstamp(1); offstamp=find(event<0); offtime=offstamp(1); mv_time=(offtime-ontime)*1/84; %sampling rate =84Hz eyes_open=data(ontime:offtime,:); %this creates a subset of data from rows that begin with event onset and end with event offset, the : means all columns are included subset of rows all columns

14. Printing Data to a File outfilename=[date,'L3_output.csv']; out_fid=fopen([outpath,outfilename],'w'); fprintf(out_fid,'filename,mn_x,std_y,mv_time,mn_x_EO\n'); fprintf(out_fid,'%s,%s,%s,\n',infilename,mn_x,SD_y,);

15. %Outputting just a String, useful for Headers. fprintf(out_fid,‘filename,mn_x, SD_y,\n'); %Outputting data from variables. fprintf(out_fid, '%s,%s,%s\n', filename,mn_x, SD_y,); Format specifier for how information is to be written, %s= string %d= decimal %f=fixed point decimal Look these up by typing “ help fprintf” Values to go in spaces L3.m

16. Special Characters you will likely use: \n = The newline character \t = The tab character \\ = The backslash character fprintf(out_fid, '%s,%s,%s\n', subj_num,group,AP_RMS); fprintf(out_fid, '%s\t%s\t%s\n', subj_num,group,AP_RMS); '\n'?

17. fprintf(out_fid, '%s,%f,%5.3f,,/n', filename, mn_x,MT,); fprintf(out_fid, '%g,%g,/n', eyes_open(1,:), eyes_open(2,:)); Output per data set One line vs. multiple lines Single variable output entire column of data from an array

18. Run the.m file 1)Save any changes to.m program 2)Type the name of the program (but not.m) in command window 3)Enter 4)Did you get any errors? 5)Did you get any messages at all? 6)Check your workspace 7)Are all your variables in place? 8)Check your desktop files 9)Did Matlab create an output folder? 10)Did Matlab create any output files? 11)Open the files 12)Do they look correct? 13)Which number specifier do you want to use? f, g, e, d?

19. Reading data with textscan infile=[inpath,infilename]; fid=fopen(infile,'r') ; if fid == -1 [infile,' could not be opened'] return end d=textscan(fid, '%s', 'delimiter', '\n'); %scans textfile and loads up as a string, delimiter chops it by each new line fclose(fid); %line above read in contents and stored it in "d", d is array of all info line by line textlines=d{1}; %this puts a wrapper on the data, begins with the first piece that is my data set

20. Creating a data array with textscan for p=2:length(textlines) herenow=strread(textlines{p},'%s','delimiter',',');%grabs the text lines and reads the whole line kind of turns it into csv with text) for q=1:length(herenow) big_array((p-1),q)=herenow(q); %(i-1) so it creates the new file from row 1 otherwise have a blank where old labels were end end

21. Creating data columns with textscan subj_num = cell2mat(big_array(:,1)); x_pos_t= str2num(cell2mat(big_array(:,2)));

22. Processing multiple data files Open L3_combine_batch_textscan.m This is a template file It is not set up to work on a current data set It has multiple inpaths to combine different files It has multiple subj_nums to allow batch processing

23. inroute='/Users/saavedra/Desktop/CP_eye_hand/key_variables/'; inroute2='/Users/saavedra/Desktop/CP_eye_hand/left_overs/'; outroute='/Users/saavedra/Desktop/CP_eye_hand/data_combine/'; if exist(outroute)==0 mkdir(outroute); end subj_num={'u11','t04','t19',};%creates an array with 1 row and a cell with a string for each subject name, you must have a folder for each subject for subj_index=1:length(subj_num)%creates an index of all the subjects inpath=cell2mat([inroute,subj_num(subj_index),'/']);% creates a string with subjnum imbedded infilename=cell2mat([subj_num(subj_index),'results.csv']); %creates an infilename with subj_num imbedded Processing multiple files

24. Counting is critical for output files Play with some common things that create weirdness in your output file 1) More headers than variables 2) Less headers than variables 3) More % s (or d or f) than variables 4) Put less %s than variables 5) Take the /n off the end of fprintf on data file Practice making errors

25. Open one of your own data files Label and sort data Keep records of the error messages Bring a record of errors to class Bring solution if you found it IF you did NOT find solution please send the error to Wayne and Sandy before 10 am Monday morning. Practice opening files