This question reared its head this week and the answer is, almost inevitably, probably you.
There are a gazillion possible scenarios, so let’s just have a look at a couple 
1 Contractor is appointed without you checking on their health and safety arrangements, competence to do the job and insurance. If there is an accident, regardless of fault, it will without doubt come down on the ‘responsible person’ – you.
2 Contractor is appointed and checked for competence, insurance and health and safety arrangements. Whilst on site, the contractor leaves a cellar hatch open in the floor and a customer falls down it. Has to be down to the contractor right? Erm, no actually. This would come down on the ‘responsible person’ – you again.
Notice the running theme? Yep; it’s always down to you. The reality is that, as the responsible person, everything that happens within the building or business that you are responsible for is going to come down to you.
The lesson is twofold:
Firstly, make sure that any contractors you appoint take their health and safety responsibilities as seriously as you do; make sure that they are insured, that their staff are trained and that they are capable of completing the task in hand safely and competently.
Secondly. When you have contractors on site be sure not to make the mistake of assuming that they will take the same amount of care as you do. Retain complete control of your business or building, and don’t be afraid of managing your contractors every bit as completely as you manage your staff, whether they like it or not.
We have to live in the real world, so what would this look like in the day to day running of a business? Well, how about a for instance?
Would I, for instance, employ a window cleaner who turned up with his ladders on a hand cart with the idea that ‘public liability’ was a medical condition and that health and safety risk assessments were for ponces?
Sadly the answer is ‘no’. The world we live in is way too litigious and for every second that that guy is on my premises I’m responsible for everything he does and everything that happens.