1. What Training is Needed by Practicing Engineers Who Create Cyberphysical Systems? Chris Scaffidi Oregon State University (USA)

2. Complexity of cyberphysical engineering Digital logic Microcontrollers Sensing Signal processing Standards Control theory Object-oriented design Middleware Circuit design Requirements Specification Teamwork Verification Validation Process & methods Architecture design System modelling Simulation and analysis

3. Need for education of many forms Traditional undergraduate, Master's degrees Lectures, laboratories, course-centric projects New opportunities for these students, and also for ongoing education of professionals Internships Summer programs Distance-learning programs Online independent learning resources Including forums

4. Online forums as useful tool for study What problems do practicing engineers ask for help with solving? What topics and what kinds of questions get answered? Where are the opportunities for training efforts? Complementary to other methods for identifying opportunities for improvements to training Give engineers a task, watch where they struggle [5] Interview experts, build consensus [16, 17] Review projects, note training necessary for success [14]

5. Study of the LabVIEW forums LabVIEW as a lens into experiences of engineers Designed with hardware-software integration in mind E.g.: sensors, analyzing signals, processing video, communicating via the internet, reading and writing databases, and controlling hardware Data obtained May 1999 – Nov 2014 150,059 forum threads 79,290 LabVIEW VI (program) attachments Including 22,187 attached to the initial questions

6. Clustering forum threads Each thread has a question, usually several responses some with attachments maybe including a marked solution. Thread content is summarized as TFIDF vector in N-dimensional space (one dimension per distinct word ever used in the forum).

7. Clustering forum threads When each thread is represented as vector, those in same cluster have high dot products. They are "in the same direction" or "near one another" in vector space.

8. Clustering forum threads Analysis to determine… What are people asking? Have topics shifted over time? Are they getting answers? Placed into 100 clusters Manually examined 10 center-most threads from each cluster (1000 total) Assigned each cluster to an overall category 14 overarching categories

9. What are people asking about?

10. Examples of what people seek Hardware I/O proportional-integral-derivative/PID controller reading and writing serial port data acquiring multi-channel data User interface fine-tune XY scatter graphs manipulate user interface panels dynamically General algorithm design how to create maintainable code how to do data logging

11. Changes over time Category 19992014Trend Hardware I/O29%22% User interface11% General algorithm design10%11% Specific LabVIEW primitives4%8% Build/deploy4%7% Signal/image/video processing5%7% Version/license/compatibility7%5% LabVIEW control structures4%6% Language interoperability10%4% Coordinating LabVIEW code4%5% File I/O4%5% Non-English2%4% Other6%3% Troubleshooting tool errors1%3%

12. Are they getting answers? Category Avg. # answer s % un- answered % un- solve d Soln. odds ratio Hardware I/O4.294%88%7 to 1 User interface4.722%83%5 to 1 General algorithm design5.144%86%6 to 1 Specific LabVIEW primitives5.182%79%4 to 1 Build/deploy4.134%86%6 to 1 Signal/image/video processing4.534%86%6 to 1 Version/license/compatibility4.553%86%6 to 1 LabVIEW control structures5.442%80%4 to 1 Language interoperability3.995%90%9 to 1 Coordinating LabVIEW code5.033%83%5 to 1 File I/O4.882%84%5 to 1 Non-English3.415%88%7 to 1 Other4.317%90%9 to 1 Troubleshooting tool errors4.433%88%7 to 1

13. Which aspects of programmers’ posts affected if they obtained solutions? Logistic regression of existing of solution versus attributes of the programmer's initial post Topic of the question Year posted Question length (characters) # VIs attached to question # Images attached # Other attachments Attached code length (KB) Explicit solution mark only supported from 2009

14. Which aspects of programmers’ posts affected if they obtained solutions? SignificantOdds impact Topic of the question up to 2.25x higher Year posted Question length # VIs attached to post 1.27x per VI # Images attached 1.10x per image # Other attachments Attached code length 0.65x per MB Overall logistic regression significant at p<0.00001; indicates significant factor at p<0.0001; based on data from 2009-14; N=72263

15. A quick look elsewhere… What other resources address engineers' problems? Used Google to search for content For each of the biggest 3 clusters in top 2 categories Manually reviewed top 5 search results => 2 * 3 * 5 = 30 results Also reviewed examples that ship with LabVIEW Only the base installation (2013) Windows Explorer

16. Key take-aways Plenty of resources found The tool vendor had resources in every search result list 1/3 rd of resources provided by 3 rd parties Extremely diverse, including code samples, videos, tutorials, and whitepapers LabVIEW included relevant examples for 5 of 6 queries But difficult (for me) to see answers or relationship among the search results No apparent emphasis on code Apparently random mixture of resources Google doesn't organize results (nor do the forums)

17. Conclusion #1: Code is Key Many topics posed by learners were software- centric, even though hardware was top topic Most topics with highest likelihood of getting answers were software-centric Posting code with a question increased the likelihood of getting a solution Many of the available online training resources are pieces of reusable code

18. Conclusion #2: Need better help finding & learning from code Not all examples are equally high in quality Google does not sort code by quality… it has no idea Relationships among resource types not obvious E.g., tracing relationships between code and other resources, such as videos and tutorials Relationships among topics not obvious How to tell what topics are "dependencies" of one another that should be mastered in a certain order? Perhaps introduce a scheme of linked case studies?

19. Thank you! To SEAA for the opportunity to present To National Instruments for funding and permission To you for your attention, questions, ideas cscaffid@eecs.oregonstate.edu