{"id":5412,"date":"2015-08-30T06:01:21","date_gmt":"2015-08-30T05:01:21","guid":{"rendered":"http:\/\/emilkirkegaard.dk\/en\/?p=5412"},"modified":"2019-04-02T09:21:31","modified_gmt":"2019-04-02T08:21:31","slug":"converting-a-data-frame-to-a-numerical-matrix-in-r-not-so-easy","status":"publish","type":"post","link":"https:\/\/emilkirkegaard.dk\/en\/2015\/08\/converting-a-data-frame-to-a-numerical-matrix-in-r-not-so-easy\/","title":{"rendered":"Converting a data.frame to a numerical matrix in R, not so easy!"},"content":{"rendered":"

Edit 2019<\/h3>\n

You don’t really need the below. Look at the `model.matrix()` function, which converts data frames into matrices for glmnet<\/strong> and similar function.<\/p>\n

Original post<\/h3>\n

Sometimes you need to use a function that wants a numeric matrix as input. One such function is glmnet.cv() which performs lasso regression with cross validation, which is very cool<\/a><\/em>. Unfortunately, it is picky about how it wants the input data. Here’s some lines of my code:<\/p>\n

fit_cv = cv.glmnet(x = as.matrix(temp_df[predictors]), #predictor vars matrix\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 y = as.matrix(temp_df[dependent]), #dep var matrix\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 weights = weights_, #weights\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 alpha = alpha_) #type of shrinkage<\/pre>\n
We see that x must be a matrix of the predictors, y must be a matrix with the dependent (usually just one), the weights and alpha are optional, but since I am working with aggregate data I am almost always using weights. Alpha controls the kind of shrinkage used.<\/p>\n
All well and good, until it isn’t. In my case, the predictor data.frame contains some factor variables. R actually uses numeric values as its internal representation of these, but displays them with strings. For instance:<\/p>\n
DF = data.frame(a = 1:3, b = letters[10:12],\r\n<\/span>                c = seq(as.Date(\"2004-01-01\"), by = \"week\", len = 3),\r\n<\/span>                stringsAsFactors = TRUE)<\/span><\/pre>\n
Which prints out like this:<\/p>\n
> <\/span>DF\r\n<\/span>  a b          c\r\n1 1 j 2004-01-01\r\n2 2 k 2004-01-08\r\n3 3 l 2004-01-15<\/pre>\n
However, suppose we use my as.matrix solution above, then we get:<\/p>\n
> <\/span>as.matrix(DF)\r\n<\/span>     a   b   c           \r\n[1,] \"1\" \"j\" \"2004-01-01\"\r\n[2,] \"2\" \"k\" \"2004-01-08\"\r\n[3,] \"3\" \"l\" \"2004-01-15\"<\/pre>\n
Which is not what we wanted. It gave us a character matrix which glmnet.cv() will then throw a nonsensical error about. Their bad error made me spend some time finding the actual error.\u00a0Save yourself and others time. Always write good error messages for functions that will be used more than a couple of times!<\/p>\n
Is there some easy built in way to solve the problem?<\/p>\n
> <\/span>as.numeric(DF)\r\n<\/span>Error: (list) object cannot be coerced to type 'double'<\/span><\/pre>\n
The easiest solution did not work.<\/p>\n
> <\/span>as.numeric(DF$b)\r\n<\/span>[1] 1 2 3<\/pre>\n
However, it does work for a single column. So maybe we can just try using it on all the columns:<\/p>\n
> <\/span>apply(DF, 2, as.numeric)\r\n<\/span>     a  b  c\r\n[1,] 1 NA NA\r\n[2,] 2 NA NA\r\n[3,] 3 NA NA\r\nWarning messages:\r\n<\/span>1: <\/span>In apply(DF, 2, as.numeric) :<\/span> NAs introduced by coercion\r\n<\/span>2: <\/span>In apply(DF, 2, as.numeric) :<\/span> NAs introduced by coercion<\/span><\/pre>\n
What? What is going on?<\/p>\n
> <\/span>apply(as.matrix(DF), 2, as.numeric)\r\n<\/span>     a  b  c\r\n[1,] 1 NA NA\r\n[2,] 2 NA NA\r\n[3,] 3 NA NA\r\nWarning messages:\r\n<\/span>1: <\/span>In apply(as.matrix(DF), 2, as.numeric) :<\/span> NAs introduced by coercion\r\n<\/span>2: <\/span>In apply(as.matrix(DF), 2, as.numeric) :<\/span> NAs introduced by coercion<\/span><\/pre>\n
It looks apply() does a silent as.matrix() which then causes the NAs. OK. How do we convert just the factor columns then? Maybe try some of the more fancy built in conversion calls:<\/p>\n
> <\/span>as.matrix.data.frame(DF)\r\n<\/span>     a   b   c           \r\n[1,] \"1\" \"j\" \"2004-01-01\"\r\n[2,] \"2\" \"k\" \"2004-01-08\"\r\n[3,] \"3\" \"l\" \"2004-01-15\"<\/pre>\n
Nope.<\/p>\n
> <\/span>as.data.frame.matrix(DF)\r\n<\/span>  a b     c\r\n1 1 j 12418\r\n2 2 k 12425\r\n3 3 l 12432<\/pre>\n
Closer, this time the date got converted, but the factor got converted to character, not integers. We could do a loop:<\/p>\n
> <\/span>for (col_idx in seq_along(DF)) {\r\n<\/span>+ <\/span>  DF[col_idx] = as.numeric(DF[[col_idx]])\r\n<\/span>+ <\/span>}\r\n<\/span>> <\/span>DF = as.matrix(DF)\r\n<\/span>> <\/span>str(DF)\r\n<\/span> num [1:3, 1:3] 1 2 3 1 2 ...\r\n - attr(*, \"dimnames\")=List of 2\r\n  ..$ : NULL\r\n  ..$ : chr [1:3] \"a\" \"b\" \"c\"<\/pre>\n
Which works, but now it is getting silly. Maybe some implicit loops:<\/p>\n
> <\/span>lapply(DF, as.numeric)\r\n<\/span>$a\r\n[1] 1 2 3\r\n$b\r\n[1] 1 2 3\r\n$c\r\n[1] 12418 12425 12432<\/pre>\n
Closer, but it returns a list, not a matrix. Maybe just try converting:<\/p>\n
> <\/span>as.matrix(lapply(DF, as.numeric))\r\n<\/span>  [,1]     \r\na Numeric,3\r\nb Numeric,3\r\nc Numeric,3<\/pre>\n
But no no, life isn’t that easy. What about as.data.frame?<\/p>\n
> <\/span>as.data.frame(lapply(DF, as.numeric))\r\n<\/span>  a b     c\r\n1 1 1 12418\r\n2 2 2 12425\r\n3 3 3 12432<\/pre>\n
Huh, that works, but as.matrix didn’t. Oh well, just one final step:<\/p>\n
> <\/span>DF = as.matrix(as.data.frame(lapply(DF, as.numeric)))\r\n<\/span>> <\/span>str(DF)\r\n<\/span> num [1:3, 1:3] 1 2 3 1 2 ...\r\n - attr(*, \"dimnames\")=List of 2\r\n  ..$ : NULL\r\n  ..$ : chr [1:3] \"a\" \"b\" \"c\"<\/pre>\n
We got what we wanted!<\/p>\n
Sometimes, R does not make your life easy.<\/p>\n","protected":false},"excerpt":{"rendered":"
Edit 2019 You don’t really need the below. Look at the `model.matrix()` function, which converts data frames into matrices for glmnet and similar function. Original post Sometimes you need to use a function that wants a numeric matrix as input. One such function is glmnet.cv() which performs lasso regression with cross validation, which is very […]<\/p>\n","protected":false},"author":17,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[2206,2208,753,2207,1979],"class_list":["post-5412","post","type-post","status-publish","format-standard","hentry","category-computer","tag-data-frame","tag-factor","tag-matrix","tag-numeric","tag-r","entry"],"_links":{"self":[{"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/posts\/5412","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/users\/17"}],"replies":[{"embeddable":true,"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/comments?post=5412"}],"version-history":[{"count":3,"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/posts\/5412\/revisions"}],"predecessor-version":[{"id":7802,"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/posts\/5412\/revisions\/7802"}],"wp:attachment":[{"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/media?parent=5412"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/categories?post=5412"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/tags?post=5412"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}