1. Dr. Borji Aisha Urooj Cecilia La Place
Sky Segmentation
Dr. Borji
Aisha Urooj
Cecilia La Place

2. Background
Sky Segmentation in the Wild: An Empirical Analysis - Mihail et al
Current vision algorithms are not effective when used on real world datasets
Quality is affected by the weather, season, and time
Using existing pretrained algorithms and methods proposed by Lu et al, Hoiem et al, and Tighe et al for their public and impact they calculated the accuracy via misclassification rate
To improve upon the prior methods’ results, an ensemble model was developed to incorporate the results and the raw images via a recurrent CNN (rCNN) model
Lu et al - weather classifier
Tighe et al - scene parser with superpixels
Hoiem et al - geometric classification of structures
Tighe was effective on 50% of images, Hoiem on 40% and Lu on 10%
This is due to time of day (sun height), month (season), weather (obscured views)
Lu and Hoiem had minimal false positive but a lot of false negative, Tighe had a lot of false positive but minimal false negative
All three combined however achieved 1.9% MCR when focusing on individual pixels

6. Goals
Run the Refinenet Cityscapes model on the cameras from the SkyFinder dataset
Develop and train a model based off Refinenet to improve current sky segmentation
Develop a model to predict the weather using either the newly trained Refinenet or a different model for sky segmentation followed by an LSTM

7. Current Results Refinenet’s Cityscapes model
45 Cameras ~ 60,000 images
mIOU %
Finetuning Refinenet
Training it on a small dataset to make minor corrections
Caffe - PSPNet in the interest of time we’re focusing on Refinenet because we were able to set that up Refinenet - had troubles with the GPU so our main focus was just running the models and then fix the GPU (now fixed)
Currently we’re working with a small subset to retrain the cityscapes model to focus on sky segmentation
After that we’ll be training the full dataset on it 5 times and averaging the error
If we still have time to consider other models or run into more issues we will be making a CNN of about 5 layers and then fully connected layers in order to connect the output of the images it to an LSTM trained on weather statistics for a possible application in weather prediction given a sequence of images

8. In Progress Steps
Familiarizing ourselves with the topic (Lit Review)
Download the datasets
Clean the data
Set up environment
Finetune RefineNet
Evaluate results
Run data on RefineNet
Future Steps
Evaluate and compare results
Propose new model
End-to-end training
Evaluate and compare results
Write paper