Scalability would mean basically some kind of increase. Maybe you can add an option of including more than one file into the program for compression in one go and that would make it scalable.
Optimizing the program for more power at less CPU cost would also mean the same in certain cases.
Modularity would mean dividing the program into sub-programs (read as: functions). So make a function for encoding, decoding, 255 chop, and so on. Just split them so that if there is a new function to be added, it can make use of certain feature of this entire RLE exclusively.
Basically speaking, divide your program into sub programs. One function for decode, another for encode and so on. And collectively call them in main() to do a task thus accomplishing 'modularity'.