Cityscapes experiment

[moyencourt]

Hi,
Thanks for open sourcing the code, this is great!
Could you share your json parameter file for cityscapes?
Also, I think it is is missing the file depth_mean.npy to be able to run it.

Thanks.

[ozansener]

We will update the code in a few days with depth_mean.npy and config files. But, in the mean time here are the config files if you do not want to wait:

• Parameter Set w/ Approximation:
  optimizer=Adam|batch_size=8|lr=0.0005|dataset=cityscapes|normalization_type=none|algorithm=mgda|use_approximation=True

• Parameter Set w/o Approximation:
  optimizer=Adam|batch_size=8|lr=0.0001|dataset=cityscapes|normalization_type=none|algorithm=mgda|use_approximation=False

depth_mean.npy is the average depth map of training set. We use it for making the input zero mean.

[moyencourt]

Thanks. Will the update code be running with pytorch 1.0? I am getting into a few problems to run it since some features are deprecated (e.g. volatile variables, .data[0], etc.)

[moyencourt]

I am also having an error with the FW step:
sol, min_norm = MinNormSolver.find_min_norm_element([grads[t] for t in tasks])

----> 1 sol, min_norm = MinNormSolver.find_min_norm_element([grads[t] for t in tasks])

~/MultiObjectiveOptimization/min_norm_solvers.py in find_min_norm_element(vecs)
     99         # Solution lying at the combination of two points
    100         dps = {}
--> 101         init_sol, dps = MinNormSolver._min_norm_2d(vecs, dps)
    102 
    103         n=len(vecs)

~/MultiObjectiveOptimization/min_norm_solvers.py in _min_norm_2d(vecs, dps)
     42                     dps[(i, j)] = 0.0
     43                     for k in range(len(vecs[i])):
---> 44                         dps[(i,j)] += torch.dot(vecs[i][k], vecs[j][k]).data[0]
     45                     dps[(j, i)] = dps[(i, j)]
     46                 if (i,i) not in dps:

RuntimeError: dot: Expected 1-D argument self, but got 4-D

What exactly should containgrads and [grads[t] for t in tasks]?

Edit: the solution is to replace with
torch.dot(vecs[i][k].view(-1), vecs[j][k].view(-1)).item()

[SimonVandenhende]

I can confirm that those changes worked for me to get the code running with pytorch 1.0.
I was able to reproduce the results for the single task models, but so far no luck with the mgda method.
Did you have to include any other changes @maria8899

[ozansener]

@r0456230 Can you tell me what exactly you are trying to reproduce? The config files I put as a comment should give exact results of mgda w/ and w/o approximation.

Please note that; we report disparity metric in paper and compute depth metric in the code. Depth map is later separately converted into disparity as post-processing. If the issue is depth, this should explain it.

mIoU should be exactly same with what reported in the code and the paper. We used the parameters I posted as a comment.

[ozansener]

@maria8899 Although we are planning to support pytorch 1.0, I am not sure when will it be. I will also update the ReadMe with the exact versions of each Python module we used. Pytorch was 0.3.1

[SimonVandenhende]

@ozansener I was able to reproduce the results from the paper for the single task models using your code (depth, instance segmentation and semantic segmentation on cityscapes).
However, when I run the code with the parameters posted above, after 50 epochs the models seems to be far removed from the results obtained in the paper.

[moyencourt]

I think I have managed to make it work with pytorch 1.0, but I still need to check the results and train it fully.
@r0456230 I haven't done much other changes, the main problem was in this FW step. Have you setup the scales/tasks correctly in the json file?

[YoungGer]

@maria8899 @r0456230 could you please tell me how to solve depth_mean.npy missing problem?

I tried the code below, but I'm not sure if it's correct

depth_mean = np.mean([depth!=0])
depth[depth!=0] = (depth[depth!=0] - depth_mean) / self.DEPTH_STD
#depth[depth!=0] = (depth[depth!=0] - self.DEPTH_MEAN[depth!=0]) / self.DEPTH_STD

[moyencourt]

@YoungGer you need to compute a mean image (per pixel) using all training images (or just a few to get an approximation) of the Cityscapes's disparity dataset.

[YoungGer]

@YoungGer you need to compute a mean image (per pixel) using all training images (or just a few to get an approximation) of the Cityscapes's disparity dataset.

I know, thank you for your help.

[JulienSiems]

@YoungGer Have you noticed, that the find_min_norm_element method actually uses the projected gradient descent method? Only find_min_norm_element_FW is the Frank-Wolfe algorithm as discussed in the paper. They are only guaranteed to be equivalent for a number of tasks equal to 2.

[kilianyp]

EDIT: Obviously I realised right after sending that question 2 is because of the optimization. Question 1 still remains.

Hi @ozansener ,
thanks for publishing your code!

I have two questions after reading this answer by @maria8899

Edit: the solution is to replace with
torch.dot(vecs[i][k].view(-1), vecs[j][k].view(-1)).item()

In your code the z variable returned by the 'backbone' of the network is passed to each task.
Its gradient is then used in the find_min_norm algorithm.

First of all as maria noted, the gradient is a 4D variable that needs to be reshaped first to 1D in torch 1.0.1.
I compared the behavior to torch 0.3.1 and it does lead to the same result, but it raised some questions, which very well might come from my missing understanding of your paper.

1. The gradient still has the batch dimension, why do you calculate the the min_norm_point between all samples as one big vector instead of for example averaging or summing over the batch dimension? Isn't this what is effectively happen after the reshaping? This is just intuitively speaking comparing it with stochastic gradient descent.
2. Why is there a batch dimension anyway? From the paper it is not quite clear to me what should be fed into the FrankWolfeSolver, but shouldn't it be the gradient of some parameters instead of an output variable? Or does that not matter and lead to the same result?

Thanks a lot!

[ozansener]

@kilsenp First let me answer the 2.

• 2: You are right if you apply MGDA directly, it should be gradients with respect to parameters. However, one of the main contributions of the paper is showing that instead you can actually feed gradients with respect to the representations. This is basically the Section 3.3 of the paper and what we are computing in the code is $\nabla_Z$.

• 1: No, you need batch dimension since forward pass of the network is different for each image. You can read Section 3.3 of the paper in detail to understand whats going on.

[liyangliu]

Hi, @maria8899, @kilsenp,
Have you reproduced the results on MultiMNIST or CityScapes? Thanks.