2. Inventor Disambiguation Workshop EVALUATION OUTCOMES

3. 7 Participant Teams U Mass Amherst IESL (USA) Centre for European Economic Research (Mannheim, DE) KU Leuven (Belgium) Penn State University (USA) InnovationPulse (USA) Centre for Transformative Innovation (CTI) at Swinburne University of Technology (Australia) Institute of Scientific and Technical Information of China (China)

4. Timeline June 2 – Training datasets posted online July 15 – ‘Intent to Participate’ deadline August 30 – Initial submission deadline – Output dataset + documentation + runtime September 1-4 – Phase 1 evaluation September 5 - Notification of progression to Phase 2 September 18 – Phase 2 submission – 2 output datasets + final documentation + AWS runtime September 18 – Phase 2 evaluation September 21 – Judges identify successful team

5. Training/test datasets Labeled datasets: human-validated inventor clusters (inventors a, b, c… on patents d, e, f… are definitely the same people) 5 datasets generously provided by: – Pierre Azoulay: de-identified academic life sciences (Azoulay 2007 and 2012) – Ivan Png LinkedIN inventors (Chunmiam et al., 2015) – Erica Fuchs, Sam Ventura: de-identified optoelectronics patents (Akinsanmi et al., 2014) – Manuel Trajtenburg: Israeli inventors (Trajtenberg and Shiff, 2008) – Francesco Lissoni, University of Bordeaux, EPO benchmark datasets * Documentation available at www.dev.patentsview.org/workshop

6. DatasetPatent-Inventor Records Unique InventorsReference OE98,762824Akinsanmi et al, 2014 ALS42,3764,801Azoulay, 2007; Azoulay, 2012 IS9,1563,845Trajtenberg and Shiff, 2008 E&S96,10414,293Chunmian et al., 2015 EPO1,922; 1,088424; 312Lissoni et al., 2010 Documentation available at http://www.dev.patentsview.org/workshop/data/README.pdf

7. Phase 1 Evaluation 5 teams successfully submitted a full output dataset of inventor clusters Output datasets were evaluated using 4 withheld labeled datasets: – Azoulay (full dataset, as it was deidentified in the training version) – Png (~20% of the data) – Trajtenberg (~20% of the data) – Azoulay – common names (a subset of the larger dataset containing only common names) Metrics calculated: – Precision, lumping, recall, splitting, F1 score Algorithm runtime also reported but with different environments, this is not a valid metric Judges reviewed the results and identified 3 teams to proceed to Phase 2 evaluation

9. TeamTest DataPrecisionRecallF1Average F1 CEERals 0.9994013470.8912683530.942242627 CEERens 10.7787271540.875600456 CEERis 0.9962459780.9222240610.957806995 CEERals_common 0.9999895750.774099570.8726635890.912078417 Innovation Pulseals 0.9917845170.6551063340.789031427 Innovation Pulseens 0.9976096420.6589726090.793679189 Innovation Pulseis 0.9983108110.6374346640.778064712 Innovation Pulseals_common 0.993635780.6381650160.7771826010.784489482 ISTICals 0.9966492480.9540906920.974905728 ISTICens 0.999478850.9217521330.959043207 ISTICis 0.9969261170.7513652160.856900212 ISTICals_common 0.9849011990.9343975210.9589848930.93745851 PSUals 0.9992970.6424640980.782102155 PSUens 10.6610015620.795907213 PSUis 0.9995780590.5880357440.740466764 PSUals_common 0.9999862960.5888889790.7412550470.764932795 UMassals 0.998880660.9763469140.987485253 UMassens 10.9661562290.982786835 UMassis 0.9988753350.9553202050.976612392 UMassals_common 0.9968851770.9633936710.9798533220.98168445 Current PatentsView Fleming/Lials 0.9990430890.8857101480.938969182 Fleming/Liens 10.8123573150.896464851 Fleming/Liis 0.9987818590.8819295050.936725547 Fleming/Lials_common 0.9980392340.8831680290.937096470.927314013

10. Phase 2 Evaluation 3 teams progressed to Phase 2: – CEER, UMass Amherst, ISTIC/China – CEER withdrew Set up identical AWS evaluation environments 2 training datasets provided – Random sample from all Phase 1 evaluation datasets – Subset of Phase 1 evaluation datasets with similar characteristics to full USPTO dataset Inventors/patents, % missing assignees, Algorithms trained and run on full USPTO dataset (twice) Algorithm documentation provided (user manual and manuscript) Output datasets evaluated with Phase 1 metrics Judges’ final determination of successful algorithm based on: – F1 score – Run time – Usability (based on documentation)

11. Team ATeam BFLEMING/Li Results when trained on random mixture dataset Results when trained on common characteristics dataset Results when trained on random mixture dataset Results when trained on common characteristics dataset No training Precision 0.9997090.9997190.9984880.9919320.999941418 Splitting 0.0339360.0333580.1168450.1038660.184882461 Recall 0.9660640.9666420.8831550.8961340.815117539 Lumping 0.0002810.0002710.0013370.0072894.78E-05 F score 0.9825990.9829030.9372870.9416030.898119352 True Positives 384367384597351380356544324310 False Negatives 1350213272464894132573559 False Positives 112108532290019 Runtime 7 hours on c3.8xlarge AWS instance N/A (CPU usage topped at 69%)(CPU usage topped at 11.85%)

12. test data precision recall Fscore Team B Results when trained on random mixture dataset eval_als 0.9981711240.9499893220.973484408 eval_als_common 0.9932018380.9208315510.955648522 eval_ens 0.9994631750.8948239720.944253473 eval_is 0.9952529940.7632798280.863966284 Results when trained on common characteristics dataset eval_als 0.9960646860.9614597020.978456322 eval_als_common 0.9823225880.9632806890.972708456 eval_ens 0.9987302340.9030746430.948496831 eval_is 0.9956739750.8379116330.910005841 Team A Results when trained on random mixture dataset eval_als 0.9988165470.9752425860.986888808 eval_als_common 0.9965893190.9597378810.977816513 eval_ens 10.9682656240.983876985 eval_is 0.9988797930.9591262620.978599468 Results when trained on common characteristics dataset eval_als 0.9988950360.9771630920.987909564 eval_als_common 0.9970359760.9664119180.981485124 eval_ens 10.9675446910.983504665 eval_is 0.9988791170.9585470790.978297585