Factories are service classes that instantiate non-injectable classes, that is, models that represent a database entity. They create a layer of abstraction between the ObjectManager and business code.
Definition of repository:
A repository object is responsible for reading and writing your object information to an object store
Use Repositories for full loading
$model->load() is not part of the service contract. I had a question on that particular topic, you might find the answers useful: Is there ever a reason to prefer
$model->load() over service contracts?
Use Factories to create new entities Repositories do not come with methods to create a new entity, so in that case you will need a factory. But use the factory for the interface, such as
Magento\Catalog\Api\Data\ProductInterfaceFactory – it will create the right implementation based on DI configuration.