The perplexity value seems to be wrong

[ylqfp]

Hi, I've just test LDA on 50w docs * 15w uniq words.
However, the result is "iteration done: 99 on 455132 samples with plexity=58.32376887781107", I'm quite confused with the value, cause it seems to be too big.

[Yunlong-He]

Hi, Ylqfp,
I feel very sorry because I don't have much knowledge about nlp, currently I use following formula to calculate the perplexity, is there anything wrong about this formula? or do you have other criteria to know how good the model is? if yes, we can implement it together.

        for (k <- 0 to m.K-1) {
          val theta: Double = (ndk(k) + m.alpha) / (words.length + m.alpha_sum)
          val phi: Double = (nwk(w)(k) + m.beta) / (nk(k) + m.beta_sum)
          l += theta * phi
        }

[ylqfp]

Hi, yunlong,
perplexity calculates total sum of the log likelihood of a word generated by one document. Thus it should be like this:

N=0
for(d=0; d<D; d++)
{
    doc_theta_vec = (ndk(k) + alpha)/(d.length + K*alpha); // should be a vector, topic distribution
    for(w=0 w<W_d; w++)
    {
        p_perword = dot(doc_theta_vec, phi[:,w]);
        log_per += -1 * log(p_perword);
    }
    N += d.length;
}
return exp(1/N * log_per)

I'll commit this weekend.

[ylqfp]

Do you know the project Zen? It's another spark-based lda, and implemented many lda features, such as sparselda/aliaslda/lightlda.
https://github.com/cloudml/zen