rush
Package website: release | dev
rush is a package designed to solve large-scale problems
asynchronously across a distributed network. Employing a database
centric model, rush enables workers to communicate tasks and their
results over a shared Redis
database. Key features include low task overhead, efficient caching, and
robust error handling. The package powers asynchronous optimization
algorithms in the bbotk
and mlr3tuning
packages.
Features
- Database centric model for robust scalability.
- Efficient communication between workers using
Redis.
- Maintains low overhead, limiting delays to just a millisecond per
task.
- Reduces read/write operations with a lightweight and efficient
caching system.
- Offers centralized system features, such as task queues.
- Provides fast data transformation from Redis to
data.table.
- Simplifies local worker setup with
processx.
- Enables scaling to large remote worker networks via the
mirai
package.
- Automatically detects and recovers from worker failures for high
reliability.
- Logs worker messages directly into the Redis database using
lgr.
- Designed with minimal dependencies for lightweight integration.
Install
Install the latest release from CRAN.
Install the development version from GitHub.
And install Redis.
- The rrq package is a
task queue system for R using Redis.
- The future
package provides a simple and uniform way of evaluating R expressions
asynchronously across a range of backends.
- batchtools is a
package for the execution of long-running tasks on high-performance
computing clusters.
- The mirai
package evaluates an R expression asynchronously in a parallel process,
locally or distributed over the network.