Would like to adopt neighbour\'s cat

Firstly, before you all accuse me of trying to steal a cat, I have 2 rescue cats who are very loyal, microchipped members of the family and who are treated like royalty!
We have a had a 6-month old cat loitering around our house for over 3 weeks now, he was running in with our cats, stealing their food and our 2 cats appeared to be inviting him in. We actively shooed him away, but over the last 2 weeks the cat has been around day and night in freezing cold weather, before the weather got cold he spent nights in our garden miaowing to come in.
I allowed him into the house and took him to the vets to be scanned, he didn't have a microchip. He was thin and dehydrated so i began to feed him and let him into the house, although he could come and go as he pleased. I put an collar on him (safety catch collar) with a note to any owner to phone me if he was their cat. I also asked neighbours if they knew of an owner, but no-one did.Although there were only 2 nights that I kept him in due to the snow, I received a text from the owner 7 days later, saying that she had taken the cat in along with his brother from a man across the street who wasn't taking care of them and she would be grateful if we wouldn't feed him. She also said that he has a tendancy to roam, which I found to be untrue as I had to force him out the door in an effort to send him home. I suggested that she should get him micro chipped as a lot of the neighbours think he is a stray.
Although we were upset as he had become part of our family, we stopped allowing him in the house and stopped feeding him immediately. He is still coming to our house and trying to come him and we have been ignoring him. However, he has been sitting outside our house miaowing all night. At 11pm last night I shooed him down the street towards his house. he was still outside at 6am this morning. I am finding it really upsetting, he is a beautiful cat who is being kicked out to spend all day in the cold weather and he really loves a warm spot to sleep. I am also concerned that he is not being fetched in at night, his home is only a few doors away, but I have never heard him being called in at night. Also, I don't think he is being fed properly as he was so thin when I finally allowed him in. He is a very submissive cat who was not at all worried about the presence of our two older cats and didn't spray at all, I am concerned that his brother is ruling the roost and he isn't getting his share of food. How can I deal with this? Ideally (and unrealisticaly!) i would like to take him in, but I would never steal someone else's cat knowing that he has a home. But I think the owner, despite caring, doesn't seem to understand that she isn't taking proper care of him by allowing him to roam all night long. Please help.