Binding different textures to descriptor sets?

When loading in a model with a lot of meshes that have different materials that contain different textures, how would you handle this in Vulkan? Is it possible to partially change a DescriptorSet with a WriteDescriptorSet object? Even if it is possible, it does not sound ideal to update the descriptor set for every mesh. I am aware of the boundless texture arrays in shader model 5.0+, but for now I want to keep it as simple as possible.