Last updated: 2023-01-01
Checks: 2 0
Knit directory: gsmash/
This reproducible R Markdown analysis was created with workflowr (version 1.7.0). The Checks tab describes the reproducibility checks that were applied when the results were created. The Past versions tab lists the development history.
Great! Since the R Markdown file has been committed to the Git repository, you know the exact version of the code that produced these results.
Great! You are using Git for version control. Tracking code development and connecting the code version to the results is critical for reproducibility.
The results in this page were generated with repository version 33d6e0e. See the Past versions tab to see a history of the changes made to the R Markdown and HTML files.
Note that you need to be careful to ensure that all relevant files for
the analysis have been committed to Git prior to generating the results
(you can use wflow_publish or
wflow_git_commit). workflowr only checks the R Markdown
file, but you know if there are other scripts or data files that it
depends on. Below is the status of the Git repository when the results
were generated:
Ignored files:
Ignored: .Rhistory
Ignored: .Rproj.user/
Untracked files:
Untracked: output/pbmc_no_constraint.rds
Unstaged changes:
Modified: analysis/index.Rmd
Note that any generated files, e.g. HTML, png, CSS, etc., are not included in this status report because it is ok for generated content to have uncommitted changes.
These are the previous versions of the repository in which changes were
made to the R Markdown
(analysis/splitting_PMF_memory_issue.Rmd) and HTML
(docs/splitting_PMF_memory_issue.html) files. If you’ve
configured a remote Git repository (see ?wflow_git_remote),
click on the hyperlinks in the table below to view the files as they
were in that past version.
| File | Version | Author | Date | Message |
|---|---|---|---|---|
| Rmd | 33d6e0e | DongyueXie | 2023-01-01 | wflow_publish("analysis/splitting_PMF_memory_issue.Rmd") |
The splitting PMF method, first implementation, requires the following matrices:
ALl matrices are dense. I only list matrices, and omit vectors.
Input: Y, S
Initialization: M
VGA: Beta(LF’), Sigma, M,V
flash: M
calc elbo: everything
As we can see, there are 6 dense matrices stored in memory all the time. The memory can easily run out on large Y, but the advantage is it’s fast because I basically traded space for time. However, for a matrix of dimension \(10^5\times 10^4\), it requires hundreds of GB of memory which is not practical.
To reduce the memory usage, I decided to do the following modification, some of which traded time for space.
A. the input matrix \(Y\) is stored in sparse format, and \(S\) is rank-1 so we only need two vectors to represent it.
B. Instead of storing Beta, I store EL, EF, two low rank matrices. When ever Beta is needed, just multiple EL,EF.
C. Sigma is not needed because sigma2 is at most a vector, Previously I formulate Sigma to simply the code.
D. For the VGA step, I did two things. First, I break samples into small folds and update each folds in a loop. So it avoids operations on huge matrices Second, since we know at optimal value, there’s an implicit relationship between M and V, we only need M. So I only stored M.
E. I also found gc() very useful to clean up the memory, but it’s very slow so I use it after each iteration.
F. do not calculate the elbo during iterations. Instead use the MSE between two sigma2 to decide if iterations should be terminated.
G. flash.fit object stores the input data M so after flash running, we can do rm(M)
However it significantly increases the computation cost because:
A. No longer support 1 iter update in VGA. (otherwise would need matrix V). (It’s fine because usually newton’s method converges in 3-5 steps)
B. Need to calculate LF’ repeatly. (Can be solving by storing LF’ for each fold)
C. Sparse matrix and dense matrix dot product is very slow. Basically element-wise operation of sparse matrix should be avoided. (Can be solving by as.matrix(Y_sparse)for each fold)
D. folds add extra for loop in vga step.
On a matrix of dimension \(10^5\times 10^3\) with \(97\%\) 0’s, the peak memory usage was reduced from 17GB to 6GB.
Maybe ultimately need a mini-batch version(like SGD) to reduce memory overload.