Association Rule Mining on Multi-Media Data Auto Annotation on Images Bhavika Patel Hau San Si Tou Juveria Kanodia Muhammad Ahmad

Auto Annotation on Images This project is on performing Association Rule Mining on Multi-relational, Multimedia Data, particularly pictures and text. Corpus: a group of 798 pictures of different kinds such as art, landscape … with descriptions Generate association rules on image data (the RGBY values), and on text data separately. Propose an algorithm to link these two different domains together. Goal: return words that will describe a given unknown picture

Offline Processing Collect 789 pictures (.jpg,.bmp) with picture descriptions Picture descriptions are saved in a file (picDescription.txt) with the format: picture description Extract keywords from the picture descriptions. Run through KeywordExtractor program to remove stop words and duplicate keywords. (keywordList.dat) Format: kewyrod1 keyword2 keyword3 Run through Apriori implementation to generate association rules Each picture is saved with the name of its unique ID. Run the pictures through a program to extract features (R, G, B, Y, orientation, intensity) values Format: R G B Y I Run the generated feature table through alterfeature program to append a unique identify for each values, as well as changing the values to relative percentage. Run through Apriori implementation to generate association rules

Multi-Arm Program Read in the 9 feature values extracted from a given image Look for all association rules in the file containing the rules on image data, with these 9 feature values as the body. Check in the feature table to find out all the pictures that have these feature values Obtain the keywords associated with each picture identified Search for all association rules in the file containing the rules on text data, with any of these keywords Output all the keywords as descriptive/related words for the given image

Association Rules on Text RulesImpliesBodySupport %Confidence % RAY<- CHANDRA2.90%87.00% CHANDRA<- RAY3.10%80.00% PAINTING<- PAINT2.50%80.00% GUIDE<- VE2.80%90.90% ARTIST<- VE2.80%90.90% PAINTING<- VE2.80%95.50% PAINTING<- COLOURS4.30%82.40% PAINTING<- GUIDE6.10%95.90% PAINTING<- ARTIST8.10%83.10% RAY<- CHANDRA IMAGE2.10%88.20% ARTIST<- VE GUIDE2.50%95.00% GUIDE<- VE ARTIST2.50%95.00% PAINTING<- VE GUIDE2.50%100.00% GUIDE<- VE PAINTING2.60%95.20% PAINTING<- VE ARTIST2.50%100.00% ARTIST<- VE PAINTING2.60%95.20% GUIDE<- FACE PAINTING2.00%81.20% ARTIST<- FACE PAINTING2.00%81.20% PAINTING<- COLOUR ARTIST2.00%93.80% ARTIST<- COLOURS GUIDE2.80%86.40% PAINTING<- COLOURS GUIDE2.80%100.00% PAINTING<- COLOURS ARTIST3.10%96.00%

Association Rules on Text RulesImpliesBodySupport %Confidence % ARTIST<- COLOURS PAINTING3.50%85.70% ARTIST<- TOP GUIDE2.30%88.90% GUIDE<- TOP ARTIST2.30%88.90% PAINTING<- TOP GUIDE2.30%100.00% GUIDE<- TOP PAINTING2.60%85.70% PAINTING<- TOP ARTIST2.30%100.00% ARTIST<- TOP PAINTING2.60%85.70% PAINTING<- WORK ARTIST2.30%83.30% PAINTING<- GUIDE ARTIST4.90%100.00% ARTIST<- GUIDE PAINTING5.90%83.00% PAINTING<- VE GUIDE ARTIST2.40%100.00% ARTIST<- VE GUIDE PAINTING2.50%95.00% GUIDE<- VE ARTIST PAINTING2.50%95.00% PAINTING<- COLOURS GUIDE ARTIST2.40%100.00% ARTIST<- COLOURS GUIDE PAINTING2.80%86.40% PAINTING<- TOP GUIDE ARTIST2.00%100.00% ARTIST<- TOP GUIDE PAINTING2.30%88.90% GUIDE<- TOP ARTIST PAINTING2.30%88.90%

Association Rules on Image Data RulesImpliesBodySupport %Confidence % 1D135<- 0B 1D02.20%82.40% 1D90<- 0B 1D02.20%94.10% 1D45<- 0B 1D02.20%88.20% 1I<- 0B 1D02.20%100.00% 1D90<- 0B 1D %85.70% 1D135<- 0B 1D902.70%85.70% 1D45<- 0B 1D %85.70% 1D135<- 0B 1D452.80%81.80% 1I<- 0B 1D %90.50% 1D45<- 0B 1D902.70%90.50% 1D90<- 0B 1D452.80%86.40% 1I<- 0B 1D902.70%100.00% 1D90<- 0B 1I3.00%87.50% 1I<- 0B 1D452.80%86.40% 1I<- 1R 1D02.40%84.20% 1D45<- 1R 1D %81.80% 1D45<- 1R 1D902.40%84.20% 1I<- 1R 1D902.40%84.20% 1D90<- 1Y 1D %80.00% 1D90<- 1D0 1D %86.90% 1D135<- 1D0 1D %86.90% 1D0<- 1D135 1D %81.60% 1D45<- 1D0 1D %83.20%

Association Rules on Image Data RulesImpliesBodySupport %Confidence % 1D135<- 1D0 1D %90.80% 1I<- 1D0 1D %90.70% 1D135<- 1D0 1I14.70%83.60% 1D0<- 1D135 1I14.80%82.90% 1D45<- 1D0 1D %82.20% 1D90<- 1D0 1D %89.80% 1D0<- 1D90 1D %80.00% 1I<- 1D0 1D %88.80% 1D90<- 1D0 1I14.70%81.90% 1I<- 1D0 1D %91.80% 1D45<- 1D135 1D %86.00% 1D90<- 1D135 1D %83.10% 1D135<- 1D90 1D %89.10% 1I<- 1D135 1D %86.00% 1D90<- 1D135 1I14.80%83.80% 1D135<- 1D90 1I15.20%81.70% 1I<- 1D135 1D %83.90% 1D45<- 1D135 1I14.80%84.60% 1D135<- 1D45 1I15.20%82.50% 1I<- 1D90 1D %89.10% 1D45<- 1D90 1I15.20%81.70% 1D90<- 1D45 1I15.20%81.70% 1D135<- 1D0 1D %90.80%

# of text association rules generated from different combination of min supp & conf

# of image association rules generated from different combination of min supp & conf

Single pass rebuild Specify common key Rebuild the tables based on the common key Use Apriori EXAMPLE: Table 1: purchase(customer,item,amount) item(customer,item_id) Table 2 purchase_total(customer,items) Query: Customers who buy a lot of stuff what do they usually but? purchase_total(X,items) return item(X,item_id)

Conclusion So we have a partial solution multimedia ARM problem, however there many things that can be done further, to improve upon it. Need to find a way to restrict the number of keywords that we get. Need to find an easier method than the present lookup method, as too many files are involved. Need for an efficient data structure to do the above point. Alternative Schemes

The End Please visit our project’s website at to find detailed information.

Questions?