[Android] I have added a lot of books and now adding books takes forever. What do I do?

In some situations creating new files on an Android device can become very slow. The most common is triggered by putting books on an SD Card, but it can also happen with books in main memory.

SD cards usually use a FAT32 file system. Directories (folders) in FAT32 are not indexed, so whenever CC creates a file the operating system (Android) must do a linear search (1st, 2nd, 3rd, 4th ... position) in order to see if the file is already there. The file is (almost) never there, so the time required is the worst case: a search of all the files in the directory. Clearly the more files in the directory the slower Android will run. For example, if there are 3,000 books in a directory, doing the check will run 3,000 times slower than when the library was empty.

A similar situation can arise with main memory for a few reasons, such as the implementation of the Android Storage Access Framework on the device being less than optimal, the device using a linear lookup file system for main memory, or the device file name cache being too small.

Regardless of the cause, limiting the number of entries in any given directory will help avoid this slowdown. You can do this by using a file save template that limits the number of entries in any one directory. We recommend using CC's file template system instead of changing the template in calibre, especially if you use the Content Server or Cloud connections. See How do I get the same file names in CC no matter which connection I use? for more information.

We ran three experiment to illustrate the problem/solution. Each experiment uses a different CC file template. The first template stores all the books in one folder, using no sub-folders (a one-level hierarchy). The second template creates a folder per first-author then stores the book (a two-level hierarchy). The third creates a folder per first letter of the first author, then a sub-folder per author (all beginning with the same letter), then the book (a three-level hierarchy).

The device is a Fire 7/rooted running CM12. The device is located next to the WiFi hub for optimal network performance.

For each experiment we cleared CC's library then sent the same 600 books to the sd card in the same order. The 600 books had 287 authors.

Results:

At 3,000 books and using the first template, the time to store a book should be approximately 16 seconds, or approximately 4 books per minute. For the last template, that time would probably still be very close to 0.65 seconds which is approximately 90 books/minute.

Notes

1. You must set CC's option Settings / File names / Use CC template everywhere if you want to use a CC template to create sub-directories. We recommend that you also set the option Replace existing book files.
   
   After you check the option you should go to book details for some book and select "Edit file name template" on the book details menu. A dialog will open showing you the current template and what file name it generates for the book you see in book details. Enter the template of your choice. See What can I use in a CC file name template? for more information about CC file name templates.
   
   
2. Simply changing the template after you have downloaded a lot of books won't fix the problem. The directories are already large. Assuming you are using the Replace existing book files option, you must resend the books already in CC's library so that they will be moved to the appropriate sub-directories. Alternatively you could reinitialize CC's library and resend all books.
   
   
3. If it is possible that you will have multiple books with identical title and authors then you must add the template item "{id }" to the end of your chosen template
   Example: {first_author:%1.1s}/{first_author}/{title} - {authors} ({id })
   NB: Remove the space after "id" and before the curly brace. The space is there only because this FAQ system's formatting rules.