It depends if the number of entries in the cache is around the number of files/folders of the external storage. There were some reports that indicate that there could be a bug somewhere with external storages:
If you removed it, check if you have still a reference in the oc_storages table. You can check the associated id, if you have entries from this storage in your oc_filecache.
you check the number of entries per storage: SELECT storage, COUNT(*) FROM oc_filecache GROUP BY storage;