Hello, I work within social care and have personal experience in my family of a person living with dementia and receiving care.
Unfortunately, due to high staff turnover and sickness it can’t be guaranteed a particular set of carers will be available. You could try contacting the care agency and asking for a small rotation of carers, perhaps two or three that can be regular carers with others filling in only when necessary.
The carers should have access to your husband’s care plan, it may be worth informing the care coordinator of your concern if you haven’t already as this could help when requesting particular carers who may work better with your husband.
A final resort is changing to a different agency hoping they may be more sympathetic, though you may be faced with other teething problems.
It is important to look after your own health and having some support from carers might seem more trouble than it’s worth, but likely to be increasingly helpful as things change in the future.