Store images efficiently in scrapy using folder structure
Introduction NOTE: This article was the reason to implement the scrapy-folder-tree scrapy extension. What is the problem and how deal with it When it comes to image storing, a common pitfall is to save all the images in a single folder. If the number of images is less than few thousands, when, stop reading this post because you will not face any issue. On the other hand, if you are planing to store numerous images, then consider splitting them in different folders. Listing a directory will become faster, more efficient and at the end of the day, your kernel will be happier. A common pattern is to create a folder structure based on the name of every file. For example, let’s say that path/to/image/dir will be the main directory, and you want to store imagefile.jpg. Create folder structure based on file’s characters and save the file inside the leaf folder: ...