Tuesday, August 14, 2012

Random and fixed effects in sensory profiling

I am reading Introduction into mixed modelling by N.W. Galway. It is partly a repeat of things I know, but I expect to use mixed models quite a lot the coming time, so it is good to repeat these things.
My problem with this book is a sensory example in chapter 2. It is profiling data, but with some twists, as happens in real life. He also makes some odd choices.

Design

The test concerns four hot products (ravioli). Because the products are hot, it was found not feasible to use a proper balanced design within each day (e.g. Williams designs). Within a day each presentation (he used presentation for rounds) has one product, as shown in the table below. Which is fine in a way, I do believe in making designs which be executed in practice. However, I come from a facility where we would at least have tried to solve this with the 'au-bain-marie'.

Table 1, allocation of products A to D over presentations and days.
Day Presentation 1 Presentation 2 Presentation 3 Presentation 4
1 B A C D
2 C D B A
3 A C B D

Within each presentation the nine assessors get served portions. It was randomized though not registered who got what serving. Even so, a random variable was created to serve as proxy. I understand this does not make a difference if servings are nested in presentations and days. Still, it removes the option of actually looking at this effect except as nested within servings and days. I can actually imagine that the product served first is different, especially in case of sloppy sensory practices, so it would be nice to be able to check this.

Model

The following effects were used in the model(s)
Table 2. Allocation of effects
Effect Allocation
Day Random
Day.Presentation Random
Day.Presentation.Serving Random
Product Fixed
Assessor Fixed
Product.Assessor Fixed

Here I got my ax to grind.

  • Product.Assessor. To quote: '(assessor) ANA perceived Brand B as the least salty, whereas (assessor) GUI perceived Brand  A as the least salty. Such crossover effects may be important: they suggest an obstacle to designing a brand that will please all consumers'. In my (industry) experience the sensory panel may well be in a different country or even continent as the target market, so that is a bit too strong. Besides, nine assessors is a bit low to segment groups on. Typically segmenting would be done with a consumer group, say 120 persons, in the target market. In my view  sensory is intended to provide an objective measurement. The assessors are representing what a typical human may taste, and are hence random. With assessors random, product.assessor can only be found random too.
  • Day.Presentation. To quote: 'Presentation 1 on Day 1 is not the the same as presentation 1 on day 2' and 'it would therefore not be meaningful to obtain the mean for presentation 1 over days'. Actually, presentation 1 is the same on day 1 as on day 2 . It is the tasting with a clean palette. While palette cleansing is supposed to make all presentations equal, that does not mean it really does. After all, this is on the trade-off between production (shorter cleaning time) and correctness (longer cleaning time). Looking at this effect is important. I would make it fixed. For round 1 specifically, it represents monadic tasting as in a consumer test. Surely a first impression is important to understand consumer liking, when correlating consumer data and sensory data I might look at solely presentation 1. Having said that, given the design in table 1, I would make presentation random. The information is not there to do anything else. 

Conclusion


I dislike the design, the sensory foundation and interpretation. However, if you ignore that sensory based dislike, the model actually makes sense and is a nice example. On top of that, the book has R code using lme (nlme package), with code on the download site http://www.wiley.com/legacy/wileychi/mixed-modelling/ has both nlme and lme4 examples, so is up to date. I am looking forward to reading the next chapters.

Wednesday, August 1, 2012

Trying Julia

In my previous post I tried building Williams designs in R. Since that code was running a bit slow, this was an ideal test for Julia. Big enough to be at least slightly realistic, small enough that it is doable.
I am very impressed. Almost twenty fold speed increase, even though this was the best I could do in R, the most naive way possible in Julia.
                                       R              Julia        Ratio
Double Williams design 4, 100 times    3.97 sec       0.212 sec    0.053   
Williams design 5, 10 times          289.5 sec       18.54  sec    0.064 


I'd surely love to use Julia more. For instance, if I could port some of the algorithm's of Professor Ng Machine Learning class to Julia? I don't think it is possible yet, but what is not, may come.

Julia code

macro timeit(ex,name,num)
    quote
        t0 = 0
        for i=1:$num
            t0 = t0 + @elapsed $ex
        end
        println("julia,", $name, ",", t0)
        #gc()
    end
end


function gendesign(ncol)
  nrow=ncol*2
  desmat = zeros(Uint8,nrow,ncol)
  desmat[1,1:ncol] = [1:ncol]
  for i = [1:ncol]
    desmat[2*i-1,1] = i
    desmat[2*i,1]=i
  end
  carover = zeros(Uint8,ncol,ncol)
  for i = 1:(ncol-1)
    carover[i,i+1] = 1
  end
  count   = 0
  addpoint( desmat , carover,count)
end

function numzero(matin) 
  length(matin) - nnz(matin) 
end   
 
function first0(matin)
   if numzero(matin)==0
      return -1
   end   
   nrow, ncol = size(matin)
   for row = 1:nrow
      for col = 1:ncol
         if matin[row,col] == 0
            return row, col
         end
      end
    end   
end   

function addpoint(desmat,carover,count)
  if nnz(desmat) == length(desmat)
     count +=1
     print("x")
     return count
  end
  row,col  = first0(desmat)
  for i = [1:size(desmat,2)]
     if numzero(desmat[row,:]-i) == 0
        if numzero(desmat[:,col]-i) < 2
           if carover[desmat[row,col-1],i] < 2 
              if (col !=2) | (desmat[row,1] != desmat[row-1,1]) | (desmat[row-1,col] < i)
                 desmat[row,col]=i
                 carover[desmat[row,col-1],i] +=1
                 count = addpoint(desmat,carover,count)
                 desmat[row,col]=0
                 carover[desmat[row,col-1],i] -=1
              end
            end 
         end 
      end 
   end 
   return count
end

@timeit gendesign(4) "design 4 " 100
@timeit gendesign(5) "design 5 " 10

Note

Both designs were created using the same script. If you ask an even number of design points using the odd algorithm, then this will work. You just get a design with double the rows, carry over balanced, every treatment equally often in each row and each column.

Tuesday, July 24, 2012

Williams designs with 5 products

In a previous post I created small Williams designs for an even number of products. This worked very well, also because the number of permutations could be restricted significantly due to symmetry. Unfortunately this does not work so well with an odd number of products. The symmetry is not so obvious. In fact, using brute force I can find non-symmetric solutions for five products. However, I am lacking force to handle more than five products in a reasonable time.

design setup

The design fixed parameters

To get the design I used a few assumptions. The number of rows is twice the number of columns. This is because I remember reading there are no solutions with the same number of rows as columns. The first row reads 1:n. The first column is all elements of 1:n twice.

Filling in the design

The approach is actually very simple.
  • Look at the first not yet know point. 
  • Examine which value it could have. 
    • not yet used in current row
    • at most once used in current column
    • at maximum once used after the product in current row, one column to the left (carry over)
    • if this is the second column and the first column of this row is the same as the first column of the previous row, then the current value must be higher than the value of one row up.
  • Try the possible values in a loop
  • Within that loop go back to step one
Implemented recursively. In all actually, it is quite brute force. I should be doing this in C or Julia rather than in R which is relatively slow. 

Tricks

Compared to the version with an even number of products I implemented some things a bit smarter. Removed some if within a for loop by selecting the objects over which the loop sequences. I also noticed some things which surprised me in making things faster. For instance, creating temporary variables (e.g. row <- desconst$todo[desobject$count,2L]) is faster than using desconst$todo[desobject$count,2L] to index matrices later on. Merging two simple lines of code into one complex line of code is slower. Apparently it is cheaper to store a small result than have the interpreter figure out a complex line of code. Hence I used quite a lot of local variables, certainly more than I find 'nice'. 

Designs

The result if 90 Williams designs for five products. Without doubt there is quite some duplication from permutations of the objects. The designs can be separated in three classes. Designs symmetric in columns (72 of those), designs with four rows symmetric in columns (five of those) and designs where are not symmetric in columns at all (13 of those).

design symmetric in columns

The first design has five unique rows each copied reversed to make the ten rows. For ease I colored the matching rows.
      [,1] [,2] [,3] [,4] [,5]
 [1,]    1    2    3    4    5
 [2,]    1    3    2    5    4
 [3,]    2    1    4    3    5
 [4,]    2    4    1    5    3
 [5,]    3    1    5    2    4
 [6,]    3    5    1    4    2
 [7,]    4    2    5    1    3
 [8,]    4    5    2    3    1
 [9,]    5    3    4    1    2
[10,]    5    4    3    2    1

designs four rows symmetric in columns

The colored rows have a reversed row in the same color. 

      [,1] [,2] [,3] [,4] [,5]
 [1,]    1    2    3    4    5
 [2,]    1    3    4    2    5
 [3,]    2    3    5    1    4
 [4,]    2    4    1    5    3
 [5,]    3    1    5    4    2
 [6,]    3    5    2    1    4
 [7,]    4    1    2    5    3
 [8,]    4    5    1    3    2
 [9,]    5    2    4    3    1
[10,]    5    4    3    2    1

design not symmetric in columns


      [,1] [,2] [,3] [,4] [,5]
 [1,]    1    2    3    4    5
 [2,]    1    3    5    2    4
 [3,]    2    4    1    5    3
 [4,]    2    5    3    4    1
 [5,]    3    1    4    2    5
 [6,]    3    2    1    5    4
 [7,]    4    3    5    1    2
 [8,]    4    5    2    1    3
 [9,]    5    1    4    3    2
[10,]    5    4    2    3    1

R code

generation

gendesign <- function(n=3) {
nc <- as.integer(n)
nr <- nc+nc
desmat <- matrix(NA,nrow=nr,ncol=nc)
desmat[1,] <- 1L:nc
desmat[,1] <- rep(1L:nc,each=2)
carover <- matrix(0L,nrow=nr,ncol=nc)
for (i in 1L:(nc-1L)) carover[i,i+1]  <- 1L
todo <- which(is.na(t(desmat)),arr.ind=TRUE)
desobject <- list(desmat=desmat,carover = carover,count=0)
desconst <- list(todo=todo,ntodo=nrow(todo),nc=nc,totest = 1L:nc)
desresult <- list()
addpoint(desobject,desresult,desconst)
}

addpoint <- function(desobject,desresult,desconst) {
desobject$count <- desobject$count+1L
if (desobject$count > desconst$ntodo) {
l <- length(desresult)
desresult[[l+1]] <- desobject$desmat
return(desresult)
}
row <- desconst$todo[desobject$count,2L]
col <- desconst$todo[desobject$count,1L]
dob <- desobject
currow <- desobject$desmat[row,]
currow <- currow[!is.na(currow)]
totest <- desconst$totest[-currow]
prev <- desobject$desmat[row,col-1L]
totest <- totest[desobject$carover[prev,totest]<2L]
counts <- rowSums(outer(totest,desobject$desmat[,col],function(x,y) as.integer(x==y)),na.rm=TRUE)
totest <- totest[counts<2L]
if (col==2 & desobject$desmat[row,1]==desobject$desmat[row-1L,1L]) {
totest <- totest[totest > (desobject$desmat[row-1L,2])]
}
for (i in totest) {
desobject$carover[prev,i] <- desobject$carover[prev,i] + 1L
desobject$desmat[row,col] <- i
desresult <- addpoint(desobject,desresult,desconst)
desobject <- dob
}
desresult
}

g5 <- gendesign(5)

checking

# check that each column has each object twice
sapply(g5,function(sub) all(sapply(1:5,function(x)   
      all(table(sub[,x])==rep(2,5)))))

# check that each product is followed by each other product twice
sub <- g5[[1]]
target <- Reduce('+',lapply(1:4,function(x) table(sub[,x],sub[,x+1])))
target  # shows a symmetric matrix with 0 on diagonal, 2 elsewhere.
sapply(g5,function(sub) identical(target,
      Reduce('+',lapply(1:4,function(x) table(sub[,x],sub[,x+1])))))

#check the number of reversed rows
sa <- sapply(g5,function(sub1) {
10-sum(1:10 %in% sapply(1:10,function(i) {
r1 <- rev(sub1[i,])
mat <- matrix(rep(r1,each=10),nrow=10,ncol=5)
which(rowSums(sub1==mat)==5)
}))
})
table(sa)
# number of reversed rows
# 0  4 10 
#13  5 72 

prints

g5[[1]]
g5[[4]]
g5[[8]]

Thursday, July 12, 2012

Creating Williams designs with even number of products

A Williams design is a special Latin square with the additional property of first order carry over (each product is followed equally often by each other product). In R the package crossdes can be used to create them.

> williams(4)
     [,1] [,2] [,3] [,4]
[1,]    1    2    4    3
[2,]    2    3    1    4
[3,]    3    4    2    1
[4,]    4    1    3    2
As a consequence of the carry over restriction, the design has the property that a row in this design is also reversed present in the design. Example, row 3 is row 1 reversed. For small designs this property can be used to generate the designs by brute force. Example; a four by four design has without loss of generality the first row and column designated 1 to 4 (using . as unknown).
1 2 3 4
2 . . .
3 . . .
4 . . .
Adding the reversal of the first row gives:
1 2 3 4
2 . . .
3 . . .
4 3 2 1
In practice, for an even number of products the last column can be created as reversed first column.
1 2 3 4
2 . . 3
3 . . 2
4 3 2 1
It is rather obvious that only one solution remains for the 2,2 location and the rest is simple filling in the blanks. The resulting design is the same as the solution of williams(4) with '3' and '4' permuted.
1 2 3 4
2 4 . 3
3 . . 2
4 3 2 1
It is possible to use the same approach for small designs with an even number of products. For an odd number of products the number of rows has to be doubled so this will follow in a later post. To create the design with 6 products a program it is more convenient than manual work. It appears, unknown to many, that using this approach there are two solutions for 6 products. These two solutions are not permutations of each other!
[[1]]
     [,1] [,2] [,3] [,4] [,5] [,6]
[1,]    1    2    3    4    5    6
[2,]    2    4    1    6    3    5
[3,]    3    1    5    2    6    4
[4,]    4    6    2    5    1    3
[5,]    5    3    6    1    4    2
[6,]    6    5    4    3    2    1

[[2]]
     [,1] [,2] [,3] [,4] [,5] [,6]
[1,]    1    2    3    4    5    6
[2,]    2    4    6    1    3    5
[3,]    3    6    2    5    1    4
[4,]    4    1    5    2    6    3
[5,]    5    3    1    6    4    2
[6,]    6    5    4    3    2    1

There are 8 solutions for 8 products and 192 solutions for 10 products. Above that, calculation time is prohibitive. Even to get to 10, the program had to be streamlined considerably.

R code

gendesign <- function(n=6) {
nr <- as.integer(n)
nc <- nr
desmat <- matrix(NA,nrow=nr,ncol=nc)
desmat[1,] <- 1L:nc
desmat[,1] <- 1L:nr
desmat[nr,] <- nc:1L
desmat[,nc] <- nr:1L
carover <- matrix(0L,nrow=nr,ncol=nc)
for (i in 1L:(nc-1L)) carover[i+1,i] <- carover[i,i+1] <- 1L
desobject <- list(desmat=desmat,carover = carover)
desresult <- list()
addpoint(desobject,desresult)
}

nextpos <- function(desmat) which(is.na(desmat),arr.ind=TRUE)

checkdes <- function(desobject,row,col) {
# test for only once carry over 
all(desobject$carover<=1) & 
# each product only once in each row and each column
!any(desobject$desmat[row,-col]== desobject$desmat[row,col],na.rm=TRUE )  &
!any(desobject$desmat[-row,col]== desobject$desmat[row,col],na.rm=TRUE )  
}

addpoint <- function(desobject,desresult) {
todo <- nextpos(desobject$desmat)
if (length(todo)==0) {
l <- length(desresult)
desresult[[l+1]] <- desobject$desmat
return(desresult)
} 
row <- todo[1L,1L]
col <- todo[1L,2L]
nc <- ncol(desobject$desmat)
dob <- desobject
for (i in 1L:nc) {
desobject$desmat[row,col] <- i
desobject$desmat[nc-row+1,nc-col+1] <- i
desobject$carover[desobject$desmat[row,col-1L],i] <- desobject$carover[desobject$desmat[row,col-1],i] + 1L
desobject$carover[i,desobject$desmat[row,col-1L]] <- desobject$carover[i,desobject$desmat[row,col-1L]] + 1L
other <- desobject$desmat[row,col+1L]
if (!is.na(other)) {
desobject$carover[other,i] <- desobject$carover[other,i] + 1
desobject$carover[i,other] <- desobject$carover[i,other] + 1
}
if (checkdes(desobject,row,col)) desresult <- addpoint(desobject,desresult)
desobject <- dob
}
desresult
}
gendesign(6)

Thursday, June 7, 2012

Simulation in the profiling model

In this post I try to make a small simulation of the sensory (flavour) profiling data, and examine if the parameters of simulated data can be retrieved by the Bayesian model build in the previous posts.
The conclusion is that it is difficult, the amount of uncertainty is too large for parts of the data. Especially, the multiplicative effect is difficult to retrieve,

Setting up the data

The data setup is the same as the data input: I will take the same design as previously used which is the chocolate data of sensominer. I will use the same kind of parameters as obtained from the chocolate data.
data(chocolates)
design <- sensochoc[,1:4]
# hyper parameters
sdPanelmean <- 0.978
sdRound <- 0.383
sdSessionPanelist <- 0.191
mu.lsdP    <- 0.387
sigma.lsdP <- 0.323
sPanelistmean <- 1.06
sPanelistsd <- 0.22
Using equivalent formulas as in the data analysis, the responses are simulated. For explanations of the terms involved I refer to the posts created in May while building the model.  I won't repeat the model here either.
nPanelist <- nlevels(sensochoc$Panelist)
nSession <-  length(unique(sensochoc$Session))
nRound = length(unique(sensochoc$Rank))
mPanelist <- rnorm(nPanelist,sd=sdPanelmean)
mround <- rnorm(nRound,sd=sdRound)
mSessionPanelist <- rnorm(nPanelist*nSession,sd=sdSessionPanelist)
meanProduct <- c(6.98,6.60,4.84,6.36,6.66,6.48)
sPanelist <- rnorm(nPanelist,mean=1,sd=0.22)
grandmean <- mean(meanProduct)
mProduct <- meanProduct-grandmean
sdPanelist <- rlnorm(nPanelist,mean=mu.lsdP,sd=sigma.lsdP)

design$score <- grandmean + mPanelist[design$Panelist] +  
 mSessionPanelist[interaction(design$Session,design$Panelist)] + 
 sPanelist[design$Panelist]*
                  (mProduct[design$Product] + mround[design$Rank])
design$score <- design$score + rnorm(nrow(design),sd=sdPanelist[design$Panelist]) 


jagsfit <- jags(data=data_list,inits=inits,model.file=model.file,parameters.to.save=parameters,n.chains=4,DIC=FALSE,n.iter=20000)

Results  

Hyperparameters

For the hyperparameters, the table shows the input data plus the resulting mean and sd of the sampler. 

input estimated mean estimated sd
sdPanelmean 0.978 0.883 0.206
sdRound 0.383 0.262 0.142
sdSessionPanelist 0.191 0.198 0.138
mu.lsdP 0.387 0.332 0.08
sigma.lsdP 0.323 0.358 0.076
It would seem from these data, that the model does reasonable, except for the two parameters involved in lsdP. To remind, these parameters are hyperparameters for the panelists individual accuracy. 

Low level parameters

The low level parameters can be extracted and plotted with a structure such as this one:
ProductMean <- fitsummary$quantiles[ 
   grep('meanProduct',rownames(fitsummary$quantiles)),]
colnames(ProductMean) <-c('x2.5','x25','x50','x75','x97.5')
ProductMean <- as.data.frame(ProductMean)
ProductMean <- cbind(meanProduct,ProductMean)
limits <- aes(ymax = ProductMean$x97.5, ymin=ProductMean$x2.5) 
p <- ggplot(ProductMean, aes(y=x50, x=meanProduct)) 
p + geom_point() + geom_errorbar(limits, width=0.2) + 
scale_y_continuous('Product mean') + 
                scale_x_continuous("Input mean") +
geom_abline(slope=1)
I feel sure people will forgive me for not repeating this scrips 4 times with different parameters.

Products mean scores

It would seem that the product means are retrieved reasonable well. 

Panelist mean scores


Panelist mean scores are retrieved reasonable well. It appears the panelists mean is estimated so accurately, that some of the parameters are different from 0. The same happened in the chocolate data, which was used to provide hyper parameters for the simulation, which seems a good thing. 

Panelist scale use (multiplicative factor)


The panelist scale is a difficult thing to estimate. On top of that, these parameters seem closer to each other than the parameters obtained from the chocolate data. The parameters also have a large error. When one thinks about this, it is not strange. Look at the product means. One product is quite different form the remainder. This means the score for this product is mainly determining a panelists scale. In regression terms, it is a leverage point. Even though every product is scored twice by each panelist, this is not good news for determining panelists scale usage. Maybe this is only possible when the products have quite different scores. But, in that case any model will do the trick of finding product differences. In case of small differences between products the model may not be performing better than a more simple model after all. 

Panelist standard error

The panelists standard error can be retrieved reasonably well. The persons with the highest values obtained do indeed have high values. It could be better, but the persons who give the highest values are indeed the worst performers. It would seem that having this parameter is useful in the model, even though the hyper parameters are difficult to estimate.

Discussion

The model performs reasonably well. However, a case can be made of removing the panelists scale use parameters. The information does not seem to be there to estimate this for the relevant data. A parameter of panelists standard error is valuable. This is a bit disappointing. I wanted a structural component, which says this panelist has the systematic behaviour, but that seems to be too difficult. What I get is a model which says a bit about quality of panelists without ability to split this into its components.